ADA051683 


^ ' February  1978 

TKCHNICAL  RKPORT  TR  77-10-SB 

On  the  Reconciliation 
of  Probability  Assessments 

D.V.  Lindley 
A.  Tversky 
R.V.  Brown 


□ecisions  ano  oesisns,  me. 


1 . ‘ 

..M  % 

« 

VUt 

l«n  taoHM  □ 

fc*;. 

OWED 

□ 

*•  ■ 

■iCAtiM 

... 

\ .... 

j >, 

J-- 

VI'IT'IS-v 

<!i!iu»!ur(  rjja 

■ti'. 

1 

TECHNICAL  RE 
■ 


lCnJrR-77- 10-25 


^ ON  THE  RECONCILIATION 
OF  PROBABILITY  ASSESSMENTS 


VW  by 

Dennis  V./indley.  Amos/^versky  and  Rex  V^own 


Sponsored  by  ' ' 
Office  of  Naval  Research] 


j Feb>wai>v  <®78 


□ecisians  ano  DesiBRs.  inc. 

Suit*  600, 8400  Westpark  Drive 
McLean,  Virginia  22101 
(703)  821-2828 

DISTRIBUTION  STATEMENT  A 

Approved  for  public  release; 

Distribution  Unlimited 


D D C 

jrairPHr::; 

MAR  99  1978 


UNCLASSIFIED 


StCUKITY  C\.»SSlflC»TtOW  OF  THIS  P»GE  0»l»  Enfrtd) 

REPORT  DOCUMENTATION  PAGE "1  beforeVompletwg^f^orm 

I REPORT  number  |2.  COVT  ACCESSION  NO.  1 RECIPIENT'S  CATALOG  NUMBER 


I REPORT  number  2.  COVT  ACCESSION  NO 

TR  77-10-25 

4.  title  Cand  Subiir/*; 

ON  THE  RECONCILIATION  OF  PROBABILITY  ASSESSMENTS 


7 autmorc*; 

Dennis  V.  Llndley,  University  College,  London 
Amos  Tversky,  Hebrew  University,  Jerusalem 

Rex  V.  Brown,  Decisions  & Designs,  Inc. 

9 performing  organization  name  and  address 
Decisions  and  Designs,  Incorporated 
8A00  Westpark  Drive 

McLean,  Virginia  22101 

11.  CONTROLLING  OFFICE  NAME  AND  ADDRESS 

Office  of  Naval  Research 
800  N.  Quincy  Street 

Arlington,  VA  22217 


5 TYPE  OF  REPORT  6 PERIOD  COVERED 


Technical 


s performing  orc  report  number 


( contract  or  grant  NUMBERr<; 


N00014-75-C-0426 


10  program  element  project,  task 

AREA  4 WORK  UNIT  NUMBERS 


12  REPORT  DATE 

Fphriiary  1Q7S 

13  NUMBER  6f  PAGES 

63 


U MONlTO^tNG  AGENCY  NAME  4 ADDRESS^//  d/ff^emnt  from  ControfUng  Offieo)  IS  SECURITY  CLASS,  (of  thf»  roport; 

UNCLASSIFIED 

“is5  DECL  ASSiriC  ATlON  DOWNGRADING 
SCHEDULE 

16  distribution  statement  (of  thU  Roporl) 

Approved  for  public  release;  distribution  unlimited. 


[Tt  distribution  statement  (of  sbtfrmct  /n  Block  20.  ff  diffororxt  from  fioport} 


10  supplementary  notes 


t9  key  words  (Continuo  on  rovortt  g/tf*  // ngcgggary  and  Idontffy  by  block  numbmr) 

Decision  Theory 
Probability  Assessment 
Coherence  Analysis 

20  ABS^^Ry^CT  (Corrtlnuo  on  rovorto  oldo  It  nocoooory  ortd  Idontlfy  by  block  numbor) 

This  paper  investigates  the  question  of  how  to  reconcile  Incoherent 
probability  assessments,  l.e.,  assessments  that  are  inconsistent  with  the 
laws  of  probal  llty.  A general  model  for  the  analysis  of  probability  assess- 
ments is  introduced,  and  two  approaches  to  the  reconciliation  problm  ar^ 
developed.  In  the  Internal  approach,  one  estimates  the  subject’s  "true 
probabilities  on  the  basis  of  his  assessments.  In  the  external  approach,  an 
external  observer  updates  his  own  coherent  probabilities  in  the  light  of  • — 


DD  y JAN^TI  1473  COITION  OF  I NOV  «l  It  OBtOLCTE 


UNCLASSIFIED 


BCCURITY  classification  of  This  pace  rPXFn  P*'*  Bnwd) 


» UNCLASSIFIED 

security  classification  of  This  PAOEfRTitn  Dmim  Enimrmd) 


' the  assessments  made  by  the  subject.  The  two  approaches  are  Illustrated 
and  discussed.  Least-squares  procedures  for  reconciliation  are  developed 
within  the  internal  approach. 


UNCLASSIFIED 

illsCCuRlTV  CLASSIFICATION  OF  THIS  RACE'MTxr.  Dmi*  tnttrrd' 


ABSTRACT 


This  paper  investigates  the  question  of  hov/  to  recon- 
cile incoherent  probability  assessments,  i.e.,  assessments 
that  are  inconsistent  with  the  laws  of  probability.  A 
general  model  for  the  analysis  of  probability  assessments  is 
introduced,  and  two  approaches  to  the  reconciliation  problem 
are  developed.  In  the  internal  approach,  one  estimates  the 
subject's  "true"  probabilities  on  the  basis  of  his  assess- 
ments. In  the  external  approach,  an  external  observer  up- 
dates his  own  coherent  probabilities  in  the  light  of  the 
assessments  made  by  the  subject.  The  two  approaches  are 
illustrated  and  discussed.  Least-squares  procedures  for 
reconciliation  are  developed  within  the  internal  approach. 


TABLES  vi 


1.0 

INTRODUCTION 

1 

2.0 

THE 

BASIC  MODEL 

9 

2.1 

Description  of  the  Model 

9 

2.2 

The  Internal  Approach 

12 

2.3 

The  External  Approach 

14 

3.0 

TECHNICAL  DEVELOPMENTS 

18 

4.0 

LEAS T-SQUARES  PROCEDURES  WITH 
INTERNAL  APPROACH 

; THE 

29 

4.1 

Partition:  General  Metric 

31 

4.2 

Sub-partition:  Probability  Metric 

35 

4.3 

External  Conditioning: 
Metric 

Probability 

36 

4.4 

External  Conditioning: 
Metric 

Log-Odds 

39 

4.5 

External  Conditioning: 

Correlations 

40 

4.6 

External  Conditioning: 

tt(X)  Assessed 

42 

4.7 

External  Conditioning: 

Causation 

44 

4.8 

Internal  Conditioning 

45 

5.0 

DISCUSSION 

49 

REFERENCES 

55 

DISTRIBUTION  LIST 

56 

TABLES 


ON  THE  RECONCILIATION  OF  PROBABILITY  ASSESSMENTS 

1.0  INTRODUCTION 

' 9 

Decisions  often  depend  on  the  probability  of  uncertain 
events  such  as  the  result  of  an  election,  the  state  of  the 
economy,  the  outbreak  of  war,  the  guilt  of  a defendant,  or 
the  outcome  of  a medical  operation.  Because  such  events  are 
essentially  unique,  the  assessment  of  their  probability  must 
be  based  on  personal  judgment.  Thus,  the  human  mind  is  used 
as  an  instrument  for  the  assessment  of  uncertainty,  much  as 
the  ruler,  the  pen  balance,  and  the  pendulum  are  used  as 
instruments  for  the  measurement  of  length,  weight,  and  time. 

In  fact,  the  axiomatic  analysis  of  subjective  probability 
(De  Finetti,  1937;  Ramsey,  1931;  Savage,  1954)  is  closely 
related  to  the  axiomatic  analysis  of  the  measurement  of 
physical  attributes  like  length,  weight,  and  time. 

* 
’ t 

To  illustrate  this  relation,  recall  that  the  analysis  ! 

of  many  physical  attributes  is  based  on  some  set  of  objects,  I 

ordered  with  respect  to  the  attribute  in  question,  and  a j 

physical  operation  of  concatenation  of  objects.  In  the  \ 

measurement  of  mass,  for  example,  the  ordering  of  objects  is 
typically  established  by  the  use  of  a pen  balance;  and  the 
concatenation  operation  is  interpreted  as  the  placing  of  two 
objects  together  in  the  same  pen.  The  key  axiom  needed  to 
establish  an  essentially  unique  additive  scale  of  mass  is 

X > y iff  xoz  > yoz 

where  > denotes  the  mass  ordering  and  o denotes  the  con- 
catenation of  objects  (see  Krantz,  Luce,  Suppes  & Tversky 
1971). 
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Suppose  that  our  object  set  consists  of  events,  and 
> denotes  the  relation  "more  probable  than"  betv?een  events. 
The  ordering  of  the  events  could  be  established  directly  by 
asking  an  individual  which  of  two  events  he  considers  more 
probable.  Alternatively,  the  ordering  could  be  derived  from 
preferences  between  gambles  as  in  the  Ramsey-Savage  approach. 
If  we  interpret  the  operation  0 as  the  union  of  disjoint 
events,  then  the  above  condition  becomes  one  of  the  critical 
axioms  needed  to  ensure  the  existence  of  a subjective  proba- 
bility measure  over  the  appropriate  collection  of  events. 

Despite  the  formal  similarity  between  the  processes, 
the  measurement  of  subjective  probability  is  considerably 
more  problematic  and  less  satisfactory  than  the  measurement 
of  physical  attributes  such  as  length  or  mass  and  psycho- 
logical attributes  such  as  loudness  or  brightness.  Indeed, 
the  assessment  of  subjective  probability  is  beset  with 
severe  problems  of  both  theoretical  and  practical  nature. 
First,  subjective  probability  is  a measure  of  degree  of 
belief,  which  reflects  one's  state  of  information.  It  is 
not  only  subjective  but  also  variable  since  it  can  change 
from  one  situation  to  another.  Second,  it  is  not  possible 
in  general  to  obtain  repeated  independent  measurements  of 
subjective  probability  from  the  same  individual  because  he 
is  likely  to  remember  his  previous  thoughts  and  responses. 
Consequently,  there  are  no  procedures  for  the  measurement  of 
belief  that  permit  the  application  of  the  law  of  large 
numbers  to  reduce  measurement  errors. 

The  difficulties  involved  in  applying  standard  measure- 
ment criteria  of  reliability  and  validity  to  the  measurement 
of  belief  give  rise  to  the  questions  of  how  to  evaluate  and 
improve  assessments  of  subjective  probability.  Three  types 
of  criteria  that  could  be  called  pragmatic,  semantic  and 
syntactic  have  been  employed.  Pragmatic  tests  refer  to 
comparisons  of  assessments  with  reality,  and  they  are 
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applicable  whenever  the  assessed  probability  of  an  event 
(e.g.,  a royal  flush  in  poker,  or  an  accident  zn  the  highway) 
can  be  meaningfully  compared  to  a value  that  is  computed  in 
accord  with  the  probability  calculus,  or  estimated  from 
empirical  data.  Unfortunately,  such  tests  cannot  be  applied 
in  most  cases  of  interest  because  of  the  difficulties  in- 
volved in  estimating  so-called  objective  probabilities. 

If  pragmatic  tests  are  not  applicable,  however,  it  is 
still  possible  to  evaluate  probability  assessments  in  terms 
of  a semantic  criterion  that  pertains  to  the  meaning  of  the 
probability  scale.  Clearly,  there  is  no  way  of  validating, 
for  example,  a meteorologist's  single  judgment  that  the  pro- 
bability of  rain  is  2/3.  If  the  meteorologist  is  using  the 
scale  properly,  however,  we  would  expect  that  rain  would 
occur  on  about  2/3  of  the  days  to  which  he  assigns  a rain 
probability  of  2/3.  This  criterion  is  called  calibration. 
Formally,  a person  is  calibrated  if  the  proportion  of  correct 
statements,  among  those  that  were  assigned  the  same  proba- 
bility, matches  the  stated  probability,  i.e.,  if  his  hit- 
rate  matches  his  confidence.  If  only  1/2  of  the  days  to 
which  the  meteorologist  assigned  a rain  probability  of  2/3 
were  rainy,  then  he  is  not  calibrated.  This  does  not  mean 
that  his  assessments  are  worthless  or  non-inf ormative ; it 
merely  indicates  an  improper  use  of  the  probability  scale. 

In  order  to  make  use  of  his  assessment,  it  should  be  remem- 
bered that  his  2/3  actually  means  1/2. 

Besides  the  pragmatic  and  the  semantic  criteria,  sub- 
jective probabilities  should  also  obey  syntactic  rules;  that 
is,  the  relations  between  assessments  should  be  governed  by 
the  laws  of  probability.  For  example,  if  A and  B are  dis- 
joint events,  then  the  assessed  probability  of  the  event,  A 
or  B,  should  be  equal  to  the  sum  of  the  assessed  proba- 
bilities for  A and  for  B.  A set  of  probability  assessments 
is  (internally)  coherent  only  if  it  is  compatible  with  the 
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probability  axioms.  Coherence  is  clearly  essential  if  we 
are  to  treat  assessments  as  probabilities  and  manipulate 
them  according  to  probabilistic  laws. 

Common  observations  and  experimental  studies  (see 
Slovic,  Fischhoff  and  Lichtenstein,  1977;  Tversky  and 
Kahneman,  1974)  show  that  the  pragmatic,  semantic  and  syn- 
tactic criteria  are  not  always  satisfied.  Thus,  assessments 
of  probability,  produced  by  laymen  and  experts  alike,  are 
often  inaccurate,  uncalibrated  and  incoherent.  But  since 
subjective  judgments  constitute  the  major  data  base  for  the 
measurement  of  uncertainty,  the  question  is  not  whether  to 
accept  subjective  judgments  at  face  value  or  reject  them 
I altogether,  but  rather  how  to  debias  and  improve  them. 

Procedures  for  the  elicitation  and  debiasing  of  subjective 
probabilities  have  been  discussed  by  several  authors,  among 
them  Kahneman  and  Tversky  (1978) ; Spetzler  and  Stael  von 
Holstein  (1975);  and  Winkler  (1969).  These  procedures  were 
designed  to  obtain  probability  assessments  that  are  more 
accurate  and  better  calibrated.  The  present  paper  is  con- 
cerned with  the  problem  of  coherence,  namely,  how  to  recon- 
cile probability  assessments  that  are  incoherent  or  mutually 
inconsistent.  Before  we  formalize  the  problem,  it  is  in- 
structive to  examine  some  examples. 

Example  1 . Consider  the  possible  causes  of  death. 
Suppose  H denotes  heart  failure,  C denotes  cancer, 

D denotes  any  other  disease,  and  N denotes  all 
causes  of  natural  death.  Let  N denote  the  complement 
of  N,  etc.  The  following  probability  assessments, 
denoted  by  q,  were  actually  made  by  one  of  the 
authors  concerning  the  possible  causes  of  death  of 
another  author : 
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■;  q(H)  = .33,  q(C)  = .27,  q(D)  = .23  q(N)  = .12. 

I 

;} 

I q(H|N)  = .41,  q(ClN)  = .31,  q(DlN)  = .28 

Note  that  these  assessements  are  incoherent.  First, 
q (H) +q (C) +q (D) +q (N)  = .95  instead  of  unity.  Such 
failures  of  additivity  are  quite  common  when  the 
number  of  events  exceeds  2 or  3.  Second,  the 
conditional  probability  ratios  do  not  coincide  with 
the  ratios  of  the  corresponding  unconditional  pro- 
babilities. For  example,  q (H j N) /q (C | N)  = 1.32, 
while  q(H)/q(C)  = 1.22.  Clearly,  the  two  ratios 
should  be  equal  because  both  H and  C are  subsets  of 
N.  The  assessor  is  now  faced  with  the  task  of 
reconciling  his  assessments  so  as  to  achieve  coherence. 
How  should  he  do  it?  What  additional  information, 
if  any,  is  required  to  reconcile  the  inconsistent 
assessments? 


Example  2.  Let  C denote  the  occurrence  of  a major 
energy  crisis  in  the  United  States  during  the  next 
decade,  and  let  q^^  and  be  two  different  assessments 
of  the  probability  of  C.  These  values  may  emerge 
from  two  different  ways  of  thinking  about  the  problem, 
say,  one  in  terms  of  specific  scenarios  that  could 
lead  to  an  energy  crisis,  and  one  in  terms  of  a 
particular  economic  model.  Alternatively,  q^  and 
q2  may  represent  the  judgments  of  two  experts  about 
the  probability  of  C.  In  general,  of  course,  q^^ 
and  q2  do  not  coincide.  Thus,  we  have  to  cimalgamate 
the  two  estimates,  that  is,  assess  the  probability  of 
C in  light  of  q^^  and  q2,  denoted  p(c|qj^,q2).  Note 
that  from  a purely  formal  viewpoint,  it  is  i^mnaterial 
whether  the  two  estimates  were  produced  by  one  person 
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using  two  different  methods,  or  by  two  different 

individuals.  In  both  cases  we  need  to  reconcile 

the  difference  and  produce  a single  estimate. 

The  problem  of  reconciling  inconsistent  observations  is 
not  unique  to  the  measurement  of  belief.  For  example,  a 
surveyor  who  uses  a theodolite  to  measure  distance  faces  a 
similar  problem.  Because  of  measurement  errors,  the  assess- 
ments of  angle  and  distance  are  generally  incompatible  with 
the  laws  of  plane  geometry.  Hence,  the  surveyor  must  recon- 
cile the  inconsistent  measurements  to  obtain  a coherent  set 
of  estimates.  His  problem,  however,  is  simpler  because  he 
can  readily  obtain  repeated  observations  and  thereby  reduce 
errors  of  measurement.  Although  it  is  generally  not  possible 
to  obtain  independent  repeated  measurements  of  subjective 
probability,  the  analogy  between  the  measurement  of  distance 
and  the  assessment  of  belief  is  instructive.  In  particular, 
it  suggests  the  possibility  of  exploiting  the  constraints 
imposed  by  the  probability  laws  to  obtain  improved  estimates 
of  subjective  probability,  much  as  the  surveyor  exploits  the 
constraints  imposed  by  plane  geometry. 

To  illustrate  this  idea,  suppose  you  have  to  assess,  as 
in  Example  2,  the  probability  of  a major  energy  crisis  in 
the  United  States  during  the  next  decade,  denoted  C.  There 
are  several  different  approaches  to  the  problem.  You  could 
adopt  an  intuitive,  wholistic  approach  where  you  contemplate 
the  energy  situation,  the  United  States  economy,  the  inter- 
national scene,  etc.,  and  make  an  intuitive  estimate  on  the 
basis  of  these  considerations.  Alternatively,  you  might 
wish  to  develop  an  explicit  model  for  the  supply  and  demand 
of  energy,  in  which  p(C)  can  be  expressed  as  a function  of 
some  parameters  that  either  are  known,  or  can  be  estimated. 

A third  possibility,  which  lies  somewhere  between  wholistic 
assessment  and  explicit  modeling,  is  to  decompose  p(C)  and 
assess  the  components  separately.  For  example,  let  E denote 


6 


an  oil  embargo  on  the  United  States  during  the  relevant  time 
period.  Following  the  decomposition  approach,  you  could 
assess  the  probability  of  an  energy  crisis  given  an  oil 
embargo  p{c|E),  the  probability  of  an  energy  crisis  in  the 
absence  of  an  oil  embargo  p(c|E),  and  the  probability  of  an 
oil  embargo  p(E).  The  overall  probability  of  an  energy 
crisis  is  given  by 

p(C)  = p(C|E)p(E)  + p(C|E)p(E),  with  p{E)  = l-p(E). 

Thus,  the  laws  of  probability  allow  us  to  compute  p(C)  from 
p(E)  and  the  conditional  probabilities  p(C|E)  and  p(C|E), 
just  as  the  laws  of  geometry  allow  us  to  compute  the  dis- 
tance between  A and  B,  say,  from  the  distance  between  A and 
X,  the  distance  between  B and  X,  and  the  angle  AXB.  Just  as 
the  distance  between  A and  B can  be  measured  using  a dif- 
ferent auxiliary  point  from  X,  the  probability  of  C can  be 
computed  using  a different  conditioning  event  from  E.  Thus, 
one  may  compute  p(C)  from  p(c|D),  p(C|D)  and  p(D),  where  D 
denotes  the  development  of  new  effective  methods  for  using 
solar  energy. 

The  use  of  different  conditioning  events  such  as  D and 
E to  compute  p(C)  can  be  viewed  as  two  different  ways  of 
thinking  about  the  target  event  C.  A direct  wholistic 
assessment  of  C represents  a third  way  of  looking  at  the 
problem.  Generally,  the  different  procedures  yield  different 
estimates  of  p(C)  that  have  to  be  reconciled.  If  each  of 
the  estimates,  however,  conveys  some  valid  information  that 
is  not  included  in  the  others,  then  the  precision  associated 
with  the  reconciled  value  will  be  greater  than  that  of 
theseparate  estimates,  in  the  same  way  that  the  precision  in 
the  location  of  a point  increases  by  determining  several 
different  (inconsistent)  bearings. 
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The  present  paper  is  concerned  with  the  development  of 
models  for  the  reconciliation  of  incoherence  (Brown  & Bindley, 
1977) . We  first  outline  a general  framework  for  the  analysis 
of  probability  assessment;  we  then  investigate  how  it  can  be 
used  to  reconcile  inconsistent  judgments.  We  develop  two 
approaches  to  the  reconciliation  problem,  which  we  label 
internal  and  external.  In  the  internal  approach,  the  observed 
probability  assessments  are  related  to  some  internal  coherent 
probabilities  in  a manner  analogous  to  the  relation  between 
the  observed  score  and  the  true  score  in  test  theory.  Thus, 
it  is  assumed  that  the  subject  has,  in  some  sense,  a set  of 
coherent  probabilities  that  are  distorted  in  the  elicitation 
process.  The  internal  approach  is  concerned,  then,  with  the 
attempt  to  estimate  the  underlying  "true"  probabilities 
using  the  observed  assessments.  This  approach  also  permits 
the  calculation  of  the  precisions  associated  with  the  recon- 
ciled values,  given  the  precisions  associated  with  the 
observed  assessments. 

In  the  external  approach,  we  introduce  an  external 
entity,  the  investigator,  who  assesses  his  own  coherent  pro- 
babilities on  the  basis  of  the  judgments  produced  by  the 
subject.  Here,  the  investigator  plays  a role  that  is  similar 
to  that  of  a surveyor  who  uses  fallible  measurements  to 
produce  a coherent  set  of  distances. 

In  the  next  section  we  develop  a general  framework  for 
the  analysis  of  fallible  probability  assessments,  and 
introduce  the  internal  and  the  external  approaches  to  the 
reconciliation  problem.  Section  3 illustrates  the  applica- 
tion of  the  basic  model  in  two  simple  special  cases.  Least- 
squares  procedures  for  the  internal  approach  are  discussed 
and  illustrated  in  Section  4.  The  philosophical  and  prac- 
tical problems  associated  with  the  present  development  are 
discussed  in  Section  5,  along  with  directions  for  future 
research. 
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2.0  THE  BASIC  MODEL 


2. 1 Description  of  the  Model 

We  are  concerned  with  an  individual  or  a subject, 
denoted  S,  who  considers  a sequence  A = (Aj^,A2, . . . ,A^)  of 
events  about  which  he  is  uncertain.  For  example,  consider 
a meteorologist  contemplating  the  rain  pattern  for  the  next 
m days  where  A^  denotes  rain  on  the  i-th  day. 

We  suppose  that  S wishes  to  describe  his  uncertainty 
about  A through  a coherent  probability  specification  for  A. 
Normative,  or  coherent,  S therefore  has  a probability  dis- 
tribution IT  (A)  for  A.  It  is  important  to  notice  that  it 
obeys  all  the  rules  of  the  probability  calculus.  In  the 
example,  the  meteorologist  would  assess  the  probabilities 
of  all  weather  patterns  over  m days,  e.g.,  ^(Aj^A^)  = ttCAj^) 

iT(A2|Aj^)  . 


Real  S is  not  necessarily  coherent  and  his  assessments 
of  probabilities  do  not  always  obey  the  rules  of  the  calcu- 
lus; even  if  they  do,  they  may  be  defective  because  of  S's 
weakness  as  a probability  appraiser.  Thus,  in  the  meteoro- 
logical situation  S may  assess  the  probability  of  rain  on 
day  1 as  0.4,  on  day  2 given  that  day  1 is  wet  as  0.8,  and 
yet  say  that  the  probability  of  rain  on  both  days  is  0.2  and 
not  0.32  as  implied  by  the  above  assessments  and  the  demands 
of  coherence.  The  assessed  values  will  be  described  by  a 
vector  q (A) . 

Our  model  therefore  contains,  in  addition  to  S,  three 
elements:  A,  it  (A)  and  q (A)  . The  first  describes  the  world 
external  to  S,  the  second  describes  a coherent  S and  the 
third  gives  S's  stated  view  of  the  world.  In  terms  of  the 
analogy  with  the  measurement  of  length,  q corresponds  to  the 
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observed  measurements  of  distances  and  angles,  and  ir  could 
be  regarded  as  the  true  distances  between  the  points. 

Like  their  physical  counterparts,  q is  directly  observable 
but  IT  is  not. 

In  addition  to  the  subject,  S,  we  consider  an  investi- 
gator, N.  Unlike  S,  N is  coherent  and  his  task  is  to  recon- 
cile S's  stated  values  q and  to  provide  an  assessment  of  it. 
Alternatively,  N could  assess  his  own  probabilities  for  A in 
the  light  of  the  information  q provided  by  S.  N can  be 
thought  of  as  the  surveyor  who  uses  Euclidean  geometry  to 
provide  estimates  of  the  true  positions  except  that  he  uses 
the  probability  calculus  instead  of  geometry.  It  is  possible 
to  interpret  N either  as  part  of  S,  or  as  external  to  S.  In 
either  case,  N is  a coherent  observer  of  S. 

From  N's  viewpoint,  all  the  elements  of  our  model,  S, 

A,  IT  and  q are  part  of  the  external  world  about  which  he  is 
uncertain  and  which  are  described  by  a probability  distri- 
bution p(A,iT,q)  of  the  uncertain  quantities.  (We  prefer 
Schlaifer's  term  "uncertain  quantity"  to  the  more  usual 
"random  variable"  because  the  expressions  are  fixed  and  not 
variable;  but  they  are  unknown  and  hence  uncertain.)  The 
notation  p(*)  will  be  reserved  for  N's  probabilities:  the 
Greek  equivalent  tt  ( • ) similarly  refers  to  S's  coherent 
probabilities;  a different  letter,  q,  is  used  for  S's 
assessments  which  need  not  be  coherent. 

The  joint  distribution  may  conveniently  be  described  in 
three  stages.  First  there  is  the  distribution  of  A,  p(A). 
Second  there  is  the  conditional  distribution  of  tt  given  A, 
P(tt|A);  and  third  there  is  the  conditional  distribution  of  q 
given  the  other  two  elements,  p(q|Tr,A).  These  three  dis- 
tributions completely  specify  the  joint  distribution  of  the 
uncertain  quantities  and  summarize  the  situation  as  far  as 
N is  concerned.  Notice  that  each  of  the  three  distributions 
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corresponds  to  a different  aspect  of  N's  contemplation  of 
the  problem.  His  view  of  the  world  external  to  both  N and  S 
is  described  by  p{A);  whereas  S's,  when  coherent,  is  ir  (A)  . 

His  view  of  coherent-S  as  a probability  appraiser  is  included 
in  p(tt|a).  Finally,  p(q|'iT,A)  gives  N's  opinion  of  S as  a 
measuring  instrument  when  he  is  measuring  his  uncertainties 
about  A.  In  the  meteorological  example,  p(Aj^)  is  N's  proba- 
bility for  rain  on  the  first  day;  p [tt  (A^^)  | A^]  is  N's  proba- 
bility that  the  meteorologist's  true  probability  of  rain  on 
the  first  day  is  it  when  it  truly  will  rain  then.  The  final 
conditional  distribution  describes  what  N thinks  the  meteor- 
ologist will  actually  state  when  his  true  value  is  it  (A, ) 

,| 

and  It  will  rain.  ii 

1 

j 

I 

Since  q differs  only  from  tt  because  of  S's  difficulties 
in  articulating  his  probabilities  and  therefore  describes  S 
as  a measuring  instrument  on  it,  it  seems  reasonable  to  suppose 
that  the  conditional  distribution  of  q,  given  tt,  does  not 
depend  on  the  state  of  the  world  external  to  S described  by 
A.  That  is,  given  tt,  q and  A are  independent,  or 

(1)  p(q|7T,A)  = p(q|7T)  . 

Indeed,  the  major  function  of  the  unobservable  "true"  proba- 
bility TT  is  to  stand  between  S and  the  external  world  so 
that  the  two  are  related  only  through  tt.  This  assumption  is 
in  the  spirit  of  the  standard  measurement  model,  where 
different  measurements  of  the  same  quantity  are  treated  as 
independent — given  the  "true"  measurement  of  the  quantity. 

In  summary,  our  model  involves 

I.  p(A);  N's  appreciation  of  the  world 

II.  p(tt|a):  N's  opinion  of  S as  a probability  appraiser 

III.  p(q|TT);  N's  opinion  of  S as  a judge  of  his  (S's) 


uncertainty  about  the  world. 


These  core  distributions  are  N's,  and  all  the  calcula- 
tions with  them  are  performed  by  N according  to  the  rules  of 
the  probability  calculus,  that  is,  coherently.  We  now  de- 
scribe two  approaches  to  the  reconciliation  problem,  called 
the  internal  and  the  external  approaches. 

In  the  internal  approach,  the  subject  acts  as  his  own 
investigator  and  attempts  to  estimate  his  "true"  coherent 
probabilities  tt.  In  the  external  approach  the  investigator 
attempts  to  revise  his  own  probabilities  for  A,  based  on 
the  assessments  made  by  the  subject. 

2 . 2 The  Internal  Approach 

Here  N is  concerned  only  with  it  and  q.  We  have^ 

(2)  p(TT)  = Z^p(tt|a)p{A) 

and  p(q|TT)  directly,  providing  a com.plete  probabilistic 
description  of  q and  tt. 

By  Bayes'  Theorem  we  have 

(3)  p(TT|q)  ® p(qlTr)p(Tr) 

as  N's  appraisal  of  coherent  S after  S has  reported  his 
assessments  q. 

This  method  can  be  used  when  S has  made  several  proba- 
bility assessments  q = (q2^,q2f*»«q  )»  finds  them  to  be 
incoherent — as  in  Example  1 or  the  meteorological  problem 
above — and  wishes  to  reconcile  them  to  coherent  values.  The 


The  abbreviated  notation  Ea  here  refers  to  a summation  over 
a partition  of  the  events  in  A.  _Thus  if  A = (Ai,A2)  then 
the  summation  is  over  A1A2 , A1A2 , A1A2  and  A1A2. 
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calculations  leading  to  (3)  enable  him  to  do  this.  Fur- 
thermore, they  provide  a way  of  calculating  the  precisions 
of  the  reconciled  values  by  means  of  the  variances  of  p(iT|q). 

Let  ('irj^,TT2, . . . .TTjjj)  be  the  true  values  that  correspond 
to  (q^ >^2 ' * * • '‘^m^  ' respectively.  There  will  typically  be 
constraints  among  the  tt^'s  corresponding  to  coherence  require- 
ments. Thus,  in  the  meteorological  problem  with  q^  = q(Aj^) 

= 0.4,  q2  = q(A2|Aj^)  = 0.8  and  = q(A^A2)  = 0.2  we  will 
have  TT^  = '^2^2*  ^ possible  set  of  reconciled  values  is  given 
by  = E(Tr.|q),  the  means  of  the  distribution  p(iT|q).  In 
statistical  language  the  it's  are  estimates  of  the  it's.  The 
precision  of  the  reconciled  value  may  be  described  by  the 
inverse  of  the  variance  of  given  q.  Notice  that,  in 
general,  the  reconciled  values  will  not  be  independent. 

A special  case  arises  when  one  or  more  of  the  events 
in  A are  of  particular  interest — we  call  them  target  events 
— and  the  other  events  are  introduced  in  order  to  increase 
the  precisions.  It  is  then  only  necessary  to  calculate  the 
marginal  distribution  of  those  it's  which  refer  to  the  target 
events.  We  call  this  procedure  extension  of  the  conversation 
— from  the  target  events  to  other  events.  The  increase  in 
precision  essentially  results  from  the  increased  exposure 
S has  to  the  constraints  of  coherence  when  he  contemplates 
many  events.  Exeunple  2 above  provides  an  illustration  of 
this  procedure.  We  believe  that  this  method  could  be  of 
considerable  value  in  improving  probability  assessments. 

The  calculations  are  described  in  detail  in  Section  4.3 
below. 
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2. 3 The  External  Approach 

Here  N is  concerned  only  with  q and  A,  so  that  S plays 

no  role  except  as  the  provider  of  data  q for  N to  update  his 

2 

probabilities  of  A.  We  have 

(4)  p(q|A)  = i^p(q|TT)p(7T|A) , 

using  the  independence  condition  (1),  and  p (A)  is  available 
directly  providing  a complete  probabilistic  description  of  q 
and  A.  By  Bayes'  Theorem  we  have 

(5)  p(A|q)  = p(q|A)p(A) 

as  N's  appraisal  of  the  uncertain  events  after  S has  reported 
his  assessments  q. 

The  meteorological  situation  provides  an  illustration 
in  which  the  meteorologist,  S,  has  announced  a probability 
of  rain  tomorrow  as  q = 0.8;  what  is  N's  probability  in  the 
light  of  this  information?  In  particular,  should  N use  S's 
stated  value?  The  external  approach  could  be  useful  where 
N is  a decision  maker  who  requires  a probability  for  A in 
order  to  take  action.  He  may  consult  an  expert,  or  a group 
of  experts,  each  of  whom  reports  his  probability  for  A and 
the  decision  maker  has  to  reach  an  overall  judgment  (see 
Excunple  2)  . 


We  use  to  denote  a summation  over  ir.  In  application  below 
it  will  be  an  integration  over  real  vector  space. 
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Notice  that  although  both  approaches  depend  on  the 
conunon  core  of  distributions  I-III  above,  p(A),  p(TrlA)  and 
p(<2|fT)»  they  can  be  used  without  N determining  all  three. 

Thus,  in  the  internal  approach,  p(tt)  may  be  assessed  directly, 
rather  than  through  (2),  when  combination  with  p(q|TT)  in 
(3)  gives  the  required  result.  Similarly,  in  the  external 
approach,  p(q|A)  may  be  assessed  directly,  rather  than 
through  (4),  and  combined  with  p(A)  in  (5)  to  produce  the 
result.  If  these  ideas  are  adopted,  the  internal  approach 
is  seen  to  be  simpler  than  the  external  one  because  it  avoids 
the  second  of  the  core  distributions,  p(tt1a),  or  the 
derived  p(q|A).  Both  these  distributions  are  relatively 
unfamiliar  in  comparison  with  I and  III,  which  are  respectively 
a "prior"  and  a likelihood.  However,  II  requires  a judgment 
by  N of  what  S believes  about  A (in  the  case  of  tt)  or  will 
say  he  believes  (for  q)  for  each  constituent  event  in  A. 

In  the  weather  example,  N has  to  consider  what  the 
meteorologist  might  say  about  rain  tomorrow  both  on  those 
occasions  when  it  will  rain,  and  on  those  when  it  will  not. 
Presiimably,  for  a reputable  weather  forecaster,  the  former 
will  be  higher  than  the  latter.  The  distribution  p(tt|a)  and 
P(tt|a)  are  measures  of  the  quality  of  the  expert,  S.  Note 
that  p(qlA)  is  related  to  the  idea  of  calibration  discussed 
earlier.  To  see  the  relation,  consider  a sequence  of 
occasions  on  which  S asserted  a value  q for  the  probability 
of  an  event  and  let  f(A|q)  denote  the  relative  frequency 
with  which  the  events  in  the  sequence  occur.  Clearly, 
f(A|q)  is  closely  related  to  p(A|q)  which,  in  turn,  is 
related  to  p(qlA)  by  the  one-dimensional  form  of  (5).  Recall 
that  a person  is  calibrated  if  f(A|q)  = q;  that  is,  if  a 
proportion  q of  the  events  to  which  he  has  assigned  a prob- 
ability q actually  occur.  Indeed,  if  p(A|q)  = q,  N will 
take  S's  announced  value,  q,  to  be  his  probability  for  A. 
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There  is  another  relationship  between  the  internal  and 
the  external  approaches.  Recall  that  their  end-products  are 
p('iT|q)  and  p(Alq),  respectively.  The  former  provides  N's 
judgment  about  S's  coherent  values,  and  the  latter  gives  N's 
assessment  of  the  external  world  in  the  light  of  S's  infor- 
mation. In  the  internal  approach,  we  proposed  that  if  = E('rTlq) 
might  be  used  as  reconciled  values  for  S's  statements.  This 
raises  the  question  of  whether  tt  (A)  = p(A|q),  that  is, 
whether  N could  use  the  reconciled  values  for  decision 
making.  To  answer  this  question  note  that 

(6)  pCAjq)  = lTTp(AlT)p{Tilq)  , 

using  the  independence  condition  (1).  This  is  ii  (A)  if 
p(Al7r)  = IT.  Hence  the  reconciled  value  will  agree  with  N's 
probability  if  N would  have  been  prepared  to  use  S's  co- 
herent probability  for  A had  he  known  it. 

We  now  add  a remark  about  the  internal  approach  in  its 

direct  form  using  p(iT)  and  p(q|TT)  to  obtain  p(TT|q).  In  this 

form,  TT  plays  the  role  of  a set  of  parameters,  q is  data, 

and  the  "prior"  p(7r)  is  updated  by  the  likelihood  pCqjTi)  to 

give  a posterior  pCulq).  The  use  of  reconciled  values  = 

E(T:|q)  with  their  associated  precisions  is  closely  related 

to  the  method  of  least  squares.  Suppose  the  data  qj,...qj^ 

are,  given  the  it's,  independent  and  normally  distributed  about 

the  it's  with  constant  variance;  suppose  further  that  the 

prior  for  the  ir's  is,  relative  to  this  likelihood,  rather 

smooth.  It  then  follows  by  standard  theory  that  the  tt's 

given  the  q's  are  also  normal.  The  means  are  the  least- 

2 

squares  estimates  obtained  by  minimizing  over  the 

it's,  subject  to  the  coherence  constraints.  (This  result 
is  exact  if  the  constraints  are  linear  and  will  be  approx- 
imate for  non-linear  constraints.)  The  matrix  of  second 
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derivatives  of  the  sum  of  squares  at  the  minimum  when 
inverted  gives  approximate  variances  and  covariances.  If 
the  q's  are  not  independent  but  have  a general  normal 
distribution,  then  a weighted  sum  of  squares  and  products 
replaces  the  direct  s\im  above.  Because  the  q's  are 
bounded,  they  cannot  be  normally  distributed  so  that  it  may 
be  preferable  to  convert  them  to  log-odds,  ln[q/(l-q)].  The 
whole  argument  then  goes  through  with  log-odds  instead  of 
probabilities,  both  for  the  q's  and  the  tt  ' s . 

This  technique  is  particularly  simple  and  is  the  most 
usable  method  we  have  for  reconciliation  of  probabilities. 

It  is  discussed  in  some  detail  in  Section  4.0.  Notice  that 

all  it  requires,  in  addition  to  S ' s provision  of  the  data,  | 

q,  are  their  variances  and  covariances  since  these  completely 
describe  the  normal  likelihood,  and  the  prior  is  assumed  to 
be  "flat."  In  our  formulation,  the  likelihood  p(qlrr)  has 
been  thought  of  as  N's  but  S may  provide  his  own  second 
moments.  If  these  are  used  directly  by  N,  our  argument  is 
unaffected . 
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3.0  TECHNICAL  DEVELOPMENTS 


In  this  section,  we  take  the  general  model  of  the 
previous  section,  insert  specific  forms  for  the  core  distri- 
butions and  calculate  other  distributions  of  interest.  To 
avoid  excessive  technicalities,  we  confine  ourselves  to  the 
very  simplest  cases  and  only  aim  to  demonstrate  the  feasi- 
bility of  the  model.  In  Section  4.0,  by  specializing  and 
using  least-squares  ideas,  we  come  nearer  to  results  of 
practical  use.  This  section  deals  with  two  special  cases  in 
which  all  the  calculations  involved  in  the  model  can  be 
displayed  explicitly. 

In  the  first  case,  there  is  a single  event.  A,  for  which 
S has  true  probability  tt{A),  or  simply  tt,  and  for  which  he 
reports  the  single  value  q(A),  or  q.  Hence  all  quantities 
are  one  dimensional.  With  one  value  reported,  there  is  no 
opportunity  to  use  coherence;  nevertheless,  reconciliation, 
in  the  sense  of  calculating  tt,  might  be  appropriate  depending 
on  N's  judgment  about  S expressed  through  the  core  distri- 
butions I-III.  In  any  case,  N may  need  to  calculate  his 
probability  of  A in  the  light  of  S's  reported  value  q,  as  is 
the  case  when  the  weather  forecaster  reports  the  probab'lity 
of  rain  tomorrow  to  be  q.  Another  example  arises  in  medical 
diagnosis:  a physician,  S,  reports  the  probability  q that 

N has  appendicitis.  What  is  N's  probability  for  appendicitis? 

To  specify  the  model  completely,  we  need  to  describe 
p(A),  p(tt|a),  p(tt|A)  and  p(qlTT).  It  is  convenient  and 
sensible  to  work  with  log-odds,  (see  above)  rather  than  with 
probabilities.  To  economize  on  notation,  we  use  q and  tt 
to  denote  log-odds  rather  than  probabilities.  The  log-odds 
for  A is  written  lo(A).  Clearly,  the  general  theory  so 


far  discussed  is  unaffected  by  this  change. 


There  are  two  reasons  for  changing  to  log-odds.  First, 
it  is  necessary  to  handle  bivariate  distributions,  and  the 
normal  is  computationally  the  most  attractive;  so  variables, 
like  log-odds,  with  infinite  ranges  are  to  be  preferred. 
Second,  it  is  more  reasonable  to  suppose  that  the  measure- 
ment error  has  constant  variance  when  expressed  in  log-odds 
rather  than  in  probabilities,  since  values  of  the  latter 
are,  in  absolute  terms,  more  precisely  assessed  when  near  0 
or  1 than  when  around  1/2 . 

We  therefore  suppose  that  the  last  of  our  core  distri- 
butions p(q|Tr)  is,  in  log-odds,  normal  with  mean  tt  and 
constant  variance,  o , abbreviated  to  N(TT,a'^).  That  is,  N 
views  S's  measurement  of  log-odds  as  unbiased  with  constant 
variance.  With  p(A)  = a,  say,  we  have  only  to  specify  p(tt1a) 
and  p(nlA).  Suppose  the  former  is  N(y^,T'^).  That  is,  if 
it  really  is  going  to  rain  tomorrow,  then  N expects  £ to 
have  log-odds  with  standard  deviation  t.  Similarly, 

suppose  the  latter  (applying  to  the  case  of  no  rain)  is 
2 

N(y2,T  ).  Presumably,  for  a good  forecaster,  y2  < 0 < y^. 
(Log-odds  of  zero  correspond  to  a probability  of  1/2.) 

A special  case  arises  when  S is  thought  to  be  just  as 
good  when  A is  true  as  when  it  is  false:  then  y2  = 
for  when  A is  true,  his  probability  for  A would  then  be 
expected  to  be  the  same  as  that  for  A when  A is  true,  but 
p(A)  = 1-p (A)  and  hence  lo (A)  = -lo(A).  This  may  be  appro- 
priate in  the  meteorological  case  but  not  in  the  appendicitis 
example,  where  it  is  easier  to  diagnose  a real  case  of 
appendicitis  than  it  is  other  sources  of  abdominal  pain,  so 
that  perhaps  |y2l<y2^.  The  variance  has  been  supposed  the 
same  for  A and  for  A;  it  would  be  possible  to  handle  the 
more  general  case,  but  the  algebra  is  untidy  and  little 
extra  insight  is  gained.  Notice  how  p(tt|A)  and  p('n|A) 
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together  describe  N's  opinion  of  5 as  a probability  ap- 
praiser. The  values  are  related  to  the  errors  of  the  two 
kinds  studied  in  statistics,  or  to  the  false  positives  and 
false  negatives  considered  in  medicine. 

2 2 

Notice  that  a and  t describe  two  quite  different 

aspects  of  variability  in  the  situation.  The  former  gives 

N's  view  of  how  far  S's  stated  value  q will  differ  from  ti, 

his  coherent  value,  and  corresponds  to  a measurement  error. 

2 

On  the  other  hand,  t describes  N's  view  of  S as  a prob- 
ability appraiser,  specifying  the  variability  in  S's  co- 
herent value  when  A is  true  (or  false) . 

In  summary,  the  model  is  as  follows: 

I,  p(A)  = a 

II,  p(Tr[A)  'V  N(p^, 

p(Tr|A)  N(u  , T^) 

2 

III,  p(q|Tr)  -v  N(tt,o^)  . 

Consider  first  the  internal  approach.  We  have,  equation  (2), 
p(Tr)  = p(tt|A)p(A)  + p(tt|A)p(A) 

(6)  = aN(p  , + (l-a)N(u  , , 

1 2 

a weighted  average  of  two  normals.  For  sufficiently  small 
2 ...  2 2 
values  of  t (if  a = U = -y  , the  condition  is  t <y  ) , 

2 1 1 

the  distribution  is  bimodal.  Thus,  good  meteorologists 
(with  small  standard  deviations)  typically  quote  probabilities 
for  rain  tomorrow  which  are  either  high  (when  it  is  going 
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to  rain)  or  low  (when  not) , and  only  rarely  are  they  so  un- 
decided as  to  give  values  around  1/2,  log-odds  of  zero. 

It  is  convenient  to  write  p(tt|a)  = P(tt|A)  = 

P^Ctt).  Then,  by  Bayes  Theorem, 

p{TTlq)  = p(q|TT)p(TT)/Z^p(qlTr)p(TT) 

p(q  (tt)  {ap^  (tt)  + (l-a)p^(Tr)} 

ap  (q)  + (l-a)p  (q) 

1 2 

= p^(TT|q)a'  + p^(iT|q)  (1-a') 

where  p^(q)  = I^P  (q  | tt)  (fr)  , p^(Ti|q)  = p (q  | rr)  p^  (it) /p^  (q) 

and  a'  = ap  (q)/{ap  (q)  + (l-o)p  (q) } . 

1 1 2 

From  standard  normal  theory,  p-(TT|q)  is  normal  with  mean 

2 2 2 ^ 22 
wq  + (l-w)vj^,  w = T /(a  +T  ) and  p^(q)  N(v^,a  +t  ),  so  that 

p(TTlq)  is  also  a mixture  of  normals.  The  mean  is 

(8)  IT  = E(TT|q)  = a'  [wq  + {l-w)iJ^]  + (1-a')  [wq  + (l-w))j^] 

= wq  + (1-w)  [a’y  + (l-a')y  ]. 

1 2 

This  is  the  reconciled  value  of  tt  on  the  basis  of  the  stated 

2 

value  q.  It  is  near  q if  w is  near  one,  that  is,  if  t is 

2 

much  greater  than  a . A large  value  of  x means  that  p(tt) 
[equation  (6)]  has  large  spread  and  that  the  likelihood 
p(q|7r)  with  smaller  spread  a is  dominant.  This  is  the 
situation  discussed  in  connection  with  least-squares  methods 
in  Section  2.3.  Consequently,  if  S has  small  measurement 
error  compared  with  appraisal  error,  the  reconciled  value 
will  be  essentially  the  stated  value,  and  no  change  will 
occur.  The  variance,  and  hence  the  precision,  of  the  recon- 
ciled value,  var{Tr|q)  can  be  found.  It  is  an  untidy 
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r 1 


2 

expression;  but  in  the  case  where  t is  much  larger  than 

2 2 
o , it  is  approximately  equal  to  o . 

2 

A nximerical  example  when  t is  of  the  same  order  as 

2 

a , so  that  reconciliation  away  from  q may  take  place,  is 
instructive.  Suppose  a = 1/2,  = 1*0  (so  that  S is 

equally  competent  whether  A is  true  or  false),  t = 1.0  and 
o = 0.5.  At  two  standard  deviations,  when  A is  true,  S 
is  anticipated  to  have  true  log-odds  between  -1.0  and  3.0 
(probabilities  between  .27  and  .95)  and  when  false  between 
-3.0  and  1.0  (.05  and  .73  for  probabilities).  But  the 
reported  odds  can  differ  by  as  much  as  1.0  (=2a)  from  the 

true  values.  (A  probability  of  0.5  can  be  reported  anywhere  J 

/s  f 

in  the  range  .27  to  .73).  Then  w = 0.8  so  that  in  it,  ’ 

equation  (8) , 80%  of  the  weight  goes  on  q and  20%  depends 
on  the  anticipated  performance  of  S.  If  q = 0.7  (stated 
probability  of  0.67),  the  weight  a'  = 0.75  and  it  is  0.66 
(a  reconciled  probability  of  0.66).  If  q = 1.5  (proba- 
bility 0.82),  the  weight  o'  = 0.92  and  tt  is  1.37  (proba- 
bility 0.80).  Both  probabilities  are  lowered  slightly,  the 
mean  of  the  prior  distribution  of  it  being  zero.  But  the 
changes  are  small,  and  even  here,  where  x is  only  twice  a, 
the  approximation  that  assumes  x is  large  is  not  unreasonable. 

In  this  case,  reconciliation  is  merely  allowing  for  the 

effect  of  measurement  error.  ; 

As  an  intermediary  between  the  internal  and  external 
approaches,  we  can  consider  p(a|it).  We  saw  just  after  j 

equation  (6)  that  N could  use  the  reconciled  value  ^ as  his  | 

probability  for  A if  p(A|ir)  = (1  + e”’^)”^,  remembering  that 
TT  is  now  a log-odds.  We  have 

p(A|Tr)  = p(Ti  |A)p(A)/p(tt) 

J 

op  (ir) 

op  (it)  + (1-a) p (tt) 

1 2 
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from  (6) . Simple  algebra  shows  the  log-odds  to  be 


(9)  lo(A|Tr)  = lo(A)  - -P^)./2t^  + (p  -p  ) ti/t  . 

12  12 

For  this  to  be  it,  two  conditions  must  be  fulfilled,  namely 

(v  -p  )/t^  = 1 and  ln-^-2 — = (p  -p  )/2t^.  These 
1 2 i-a  1 2 

are  equivalently 


(10) 


(P^-P2)A^  = 1 and  l/2(pj^+p2) 


The  second  condition  says  that  the  prior  log-odds  have  to 

equal  the  average  performance.  This  is  satisfied  in  the 

special  case  p = -p  when  a = *5.  The  first  condition  says 
2 1 

that  the  difference  between  the  expected  performances  under 

the  two  conditions,  A and  A,  must  equal  the  appraisal  variance 
2 

T . Thus,  a complete  matching  of  N and  coherent-S  when 

considering  A depends  on  a rather  complex  combination  of 

prior  views  by  N and  S's  appraisals.  A change  in  the 

former  would  affect  this  agreement.  If  N judges  A to  be  as 

likely  as  not,  a = Hi  the  conditions  for  agreement  are  that 

p = -p  , so  that  S performs  equally  well  under  A as  A,  and 
2 1 

2p  /t^  = X,  say  is  one.  No  simple  interpretation  of  this 
final  requirement  is  known  to  us. 

Consider  next  the  external  approach.  It  is  immediate 
from  the  core  distributions  II  and  III  that 

(11)  P(q|A)  'V  N(p  , 0 +T^)  and  p(qlA)  'v-  N(p  ,o^+t*), 

1 2 

so  that  the  replacemcuit  of  it  (in  II)  by  q here  merely 
replaces  t*  by  o^+x^.  Consequently,  p(Alq)  is  given  by 
the  same  result  as  (9)  but  with  x*  everywhere  replaced  by 
o*+x*.  The  "least-squares'*  case  where  x*  is  large  com- 
pared with  makes  the  two  quantities  p(A|iT)  and  p(A|q) 
nearly  equal. 
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In  the  numerical  illustration  with  a = 1/2,  y = y 

1 2 

= 1.0,  T = 1.0  and  o = 0.5,  so  that  X = 2 , the  log-odds 

given  q are,  from  (9)  with  o*+t^  replacing  and  q for 

IT,  equal  to  (y  -y  )q/(o*+T^),  or  1.6  times  q.  Hence  q of 
1 2 

0.7  (probability  0.67)  is  raised  to  q of  1.12  (probability 

0.75)  and  a q of  1.5  (probability  0.82)  to  one  of  2.4 

(probability  0.92).  These  changes  are  much  larger  than 

those  caused  by  reconciliation.  The  intuitive  explanation 

for  this  is  most  easily  seen  by  considering  the  higher  value 

of  q,  1.5.  Such  a value  is  much  more  likely  to  have  arisen 

from  the  distribution  when  A is  true,  p^  (q)  N(l,1.25) 

rather  than  from  the  distribution  when  A is  false,  p (q)  'v- 

2 

N(-l,  1.25).  This  is  described  by  o'  which  is  here  0.92,  so  f 

that  there  is  a 92%  chance  that  q came  from  A.  Consequently, 

N can  substantially  increase  S's  stated  value.  Notice  how 
this  change  depends  heavily  on  S as  a probability  appraiser 
whereas  reconciliation  does  not,  p(tt|a)  and  p(tt|A)  enter 
only  through  p(tt).  If  t increases,  the  effect  on  N will 
become  less.  In  an  extreme  case  where  t and  o tend  to  zero, 

N will  shift  S's  stated  value  to  either  1 or  0 . A meter- 
ologist  who  always  says  0.6  chance  of  rain  on  rainy  days 
and  0.4  when  it  is  to  be  dry  entitles  N to  be  sure  of  rain 
when  0.6  is  announced. 

The  second  example  differs  from  the  first  only  in 
that  S assesses  both  the  probability  of  A and  of  A,  supplying 
q(A)  and  q(A).  These  we  denote  by  q^  and  q respectively. 

There  is  now  a possiblity  for  incoherence  since  they  may  ! 

not  add  to  1.  With  rather  general  normal  distributions,  the 
algebra  becomes  complicated;  but  it  is  possible  to  demon- 
strate the  main  points  if  we  specialize  somewhat.  Specifi- 
cally, we  suppose  (the  arguments  still  being  log-odds) 
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I 


I: 

p(A)  = 

h, 

2 

II: 

p(TT  1 A) 

N ( u , T ) 

2 . 

p (tt  1 A) 

'T'  K ( - y » T ) 

(these  are 

as  before 

with  a = ^ and  y = -y  ) 

2 1 


For  the  last  stage,  we  need  to  describe  the  joint  distribution 
of  and  q^.  This  we  suppose: 

III:  p(q^|iT)  'V/  N(TT,a  ) and 

p(q  I tt)  N(-TT,a^)  , 

2 

these  being  independent,  given  tt. 

The  last  condition  says  that  both  q and  q = q(A) 

1 2 

are  unbiased,  of  equal  precision  and  are  uncorrelated. 

This  imples  that,  for  all  tt,  E(q  +q  |tt)  =0,  so  that  N 

1 2 

expects  S to  be  coherent.  A generali-zation  would  allow  for 

bias.  Another  generalization  would  permit  q and  q to  be 

1 2 

correlated.  For  example,  S may  always  have  q +q  = 0,  in 

1 2 

which  case  they  are  perfectly  negatively  correlated  and  we 
are  back  to  the  first  example. 

In  the  internal  approach,  the  calculation  of  p(tt)  is  as 

before,  equation  (6) , with  its  possible  bimodal  form.  The 

revised  value  of  this  distribution  p(Tr|q  ,q  ) is  of  the 

1 2 

seime  character  as  before,  equation  (7)  ; a mixture  of  two 

normal  distributions  separately  derived  from  p^  (tt)  * p(tt|a) 

and  (tt)  = p(tt|a).  Standard  normal  theory  calculations 

show  that  p (irlq  ,q  ) has  mean 
1 12 


2'^ 


(q  -q  ) y 

/ 

2 1 

1 * + 

/ 

+ 

2 2 
a T 

1/ 

2 2 
a T 

2 2 • ^ 

with  variance  {2'/a  + 1/t  } : and  that  (it  |q^  ,q^)  is  of 

the  same  form  with  -y  for  y.  The  weights  are 

w'  = p (q  ,q  )/{p  (q  ,q  ) + p (q  ,q  ) } and  1-w' . Thus  the 
112  112  2 12 

reconciled  value  is 

(12)  IT  = E(TTlq^,q^)  = (q^-  q^)/o^  y(2w'  -1)/t^ 

+ • 

2/a^  + 1/t^  2/a^  + 1/t^ 

2 2 

For  the  case  where  t is  large  compared  with  o , this  is 

about  J5(q^  - • This  simple  form  is  interesting:  it  is 

’sCq  + (-q  )},  the  average  of  q and  (-q  ) which  are  effectively 
12  12 

two  statements  made  by  S about  A,  and  not  one  about  A and 
one  about  A.  The  first  is  direct;  the  second  was  originally 
q for  A,  so  -q  for  A if  coherent. 

A numerical  example  may  prove  illuminating.  Suppose  S 
gave  probabilities  of  0.6  for  A and  0.3  for  A.  The  correspond- 
ing log-odds  are  q =0.41  and  q = -0.85.  Then  1/2  (q  -q  ) 

1 2 12 
= 0.63,  giving  a reconciled  value  for  S's  coherent  probability 

for  A of  0.65,  and  of  course  0.35  for  A. 

The  variance  of  the  reconciled  value  of  the  log-odds 
2 2 

is  Ho  , for  large  t ; one  half  the  variance,  twice  the 
precision,  of  the  separate  assessments.  Suppose,  in 
illustration,  a = 0.2,  so  that  at  two  standard  deviations 
the  log-odds  for  A could  lie  in  the  range  0.41  + 0.4,  or 
in  probability  terms  (0.50,0.69).  Similarly,  the  range  of 
probabilities  for  A is  (0.22,  0.39).  The  consequent  range 
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for  the  reconciled  probability  of  A,  tt  is  (0.59,  0.71). 

The  calculation  of  p(A|tt)  is  as  before  with  no 

2 

alteration,  and  (9)  simplifies  to  give  1o(A|it)  = 2vnT/T 
= XiT  in  the  earlier  notation. 

For  the  external  approach,  we  have  first  to  combine 

II  and  III  to  obtain  p(q  ,q  |A)  and  p(q  ,q  |A).  The  former 

12  12 

is  clearly  a bivariate  normal  distribution  with  means 

2 2 2 

(y,-u) , variances  a + t and  covariance  -t  . Conditional 
on  A,  the  distribution  is  the  same  with  -y  for  y.  Direct 
calculation  using  Bayes  Theorem  shows  that 

(13)  lo(Alq  ,q  ) = 2y (q  -q  )/(a^+2T  ) 

12  12 

2 2 *^2 

If  T is  large  in  comparison  with  a , this  is  about  2yTT/T  . 

If  T = 1.0  (taken  as  large  in  comparison  with  a = 0.2)  and 

y = 1.0,  our  numerical  illustration  gives  a value  for  the 

log-odds  of  1.20,  corresponding  to  a probability  of  0.77. 

This  is  a substantial  increase  from  a stated  probability  of 

0.6,  reconciled  to  0.65  to  allow  for  incoherence.  The 

reason  is,  as  in  the  first  example,  that  N judges  that  the 

stated  values,  0.6  and  0.3,  have  more  likely  arisen  from 

A,  than  A,  so  that  he  increases  the  value  for  p(A|q  ,q  ). 

1 2 

These  two  examples  demonstrate  how  the  ideas  work  out 
in  simple  cases  and  show  that  the  internal  approach  is 
simpler  than  the  external  one,  particularly  when  least-squares 
procedures  are  used.  The  difficulty  with  the  external 
approach  lies  in  the  use  of  the  measures  of  S's  ability  as 
a probability  appraiser,  p(tt|a),  which  do  not  enter  into 
the  other  apparoach.  Nevertheless,  there  are  problems  where 
the  external  approach  is  essential,  as  when  N has  separate 
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opinions  from  several  experts  and  needs  to  use  them  to 
assess  his  own  probability  for  A.  We  pass  in  the  next 
section  to  a consideration  of  the  internal  approach  using 
least  squares. 
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4.0  LEAST-SQUARES  PROCEDURES 
WITH  THE  INTERNAL  APPROACH 


We  begin  with  a general  description  of  the  procedure. 

We  are  concerned  only  with  q and  rr,  S's  stated  and  coherent 
values  for  some  events.  N's  opinion  about  "n  is  diffuse  so 
that  p(tt)  is  approximately  constant.  In  the  simpler  situa- 
tions, each  element  of  q is  approximately  normally  dis- 
tributed about  TT^,  the  corresponding  element  of  ir,  with 
constant  variance  say,  these  being  independent.  Under 
these  circumstances,  the  reconciled  values  = ECir^i^lq) 
are  given  approximately  by  the  values  of  the  it's  that  minimize 

subject  to  any  constraints  on  the  ir's  that  coherence  imposes. 
In  other  situations,  the  q's  will  be  correlated  and  have 
different  variances,  when  a quadratic  form 

Zw.  . (q.-TT . ) (q  .-IT  . ) 

will  be  minimized  subject  to  the  constraints,  the  w's  being 
weights,  the  elements  in  the  inverse  of  the  dispersion  matrix 
of  the  q's.  Minimization  is  performed  by  equating  the  first 
derivatives  to  zero.  The  matrix  of  second  derivatives  at 
the  minimum,  when  inverted,  provides  the  approximate  vari- 
ances, var(TTj^lq),  and  covariances  of  the  reconciled  values. 
Throughout  this  argument,  the  n's  and  q's  may  be  proba- 
bilities or  some  suitable  transform  of  them,  such  as  log- 
odds  . 


The  whole  procedure  is  straightforward  except  for  one 
difficulty:  the  constraints  imposed  by  coherence  are 
typically  non-linear,  and  the  resulting  equations  are  there- 
fore non-linear.  There  is  no  simple  resolution  of  this 
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difficulty.  The  power  of  the  probability  calculus  lies  in 
the  ability  of  probabilities  to  combine  both  additively  and 
multiplicatively . If  linearity  of  the  latter  is  imposed  by 
taking  logarithms,  the  linearity  of  the  former  is  destroyed. 
It  is,  therefore,  a fundamental  difficulty  but  one  that  can 
sometimes  be  alleviated  by  suitable  approximations. 


The  coherent  probabilities  TT^(l<i<n)  will  themselves  be 
functions  of  k<n  probabilities  that  can  each  coherently 
receive  any  value  in  the  unit  interval  irrespective  of  the 
other.  For  example,  with  two  events,  A^,,  A2,  where  S has 
provided  = q(A^)  , q2  = q(A2lAj^)  and  q^  = q{A^A^)  , then  if 

= it(Aj^)  and  62  = tt(A21a^),  we  have  ''^1  = 

^3  ~ ®1®2  ® “ 2,  n = 3.  If  k <n,  then 

coherence  imposes  constraints  on  the  tt's.  Denote,  as  in  the 
example,  these  basic  probabilities  by  6^^,  62,... Then 
confining  attention  to  the  uncorrelated,  equal  weights  case 
(generalizations  being  clear) , the  expression  to  be  minimized 
with  respect  to  the  6 ' s is 

(14)  i[q^  - iT^(e^,e2,...ej^))^. 


Differentiation  with  respect  to  6^  gives 


(15)  Z^[q^  - Tr^(e)]  3tt^/8^  = Q . 

The  usual  way  to  solve  these  equations  in  the  non-linear 

A 

case  is  to  suppose  we  have  some  first  guess  6^  at  6 — for 
example,  the  values  suggested  by  qj^,q2, . . .qj^ — and  expand 
iT^(e)  in  a Taylor  series  about  0^  retaining  only  the  first 
term.  The  resulting  equations  are  linear  in  the  discrepancy 
and  may  therefore  be  solved. 


Numerically  these  procedures  may  be  iterated  to  obtain 
improved  solutions.  In  what  follows,  we  shall  concentrate 

1 

I 
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on  analytical  solutions,  either  exact  or  approximate,  which 
help  us  to  understand  the  methods,  recognizing  that  for 
practical  implementation  in  complex  situations,  computer 
software  will  have  to  be  provided.  This  provision  should 
not  be  difficult  since  many  minimization  procedures  for  non- 
linear functions  are  available. 

We  now  pass  to  the  consideration  of  several  special 
cases.  In  any  application  of  the  least-squares  ideas,  we 
have  to  specify  whether  the  calculation  is  in  terms  of 
probabilities  directly,  or  in  some  function  thereof,  such  as 
log-odds.  We  refer  to  this  as  the  choice  of  metric:  proba- 
bility metric,  log-odds  metric,  etc.  The  reason  for  pre- 
ferring one  metric  to  another  is,  as  before,  that  the  variance 
of  the  q's  may  be  judged  to  be  more  reasonably  constant  in 
one  metric  than  in  another. 

4 . 1 Partition:  General  Metric 

Here  (A^^, A2 , . . .A^)  is  a partition,  and  S provides 
probabilities  q.  = q(A. )»  l<i<n  for  it.  = tt(A.)  with  the 
single'*’  coherence  constraint  Itt^=  1.  For  a general  metric 
F(.)  — for  log-odds,  F(t)  = ln[t/(l-t)]  - the  expression  to 
be  minimized  is  E[F(q^)  - F(tt^)]2  subject  to  the  constraint 
Itt^  = 1.  The  constraint  can  either  be  incorporated  as 
above,  with  0.  = tt  . for  i<n  and  ti  = 1 - or  by  using 

a Lagrangian.  For  the  probability  metric,  the  equations 
are  linear  and  have  the  exact  solution 

(16)  T?^-q^  = n "^(l-lq^) 
with 

(17)  var(^^)  = (l-n"^)o^. 


^It  is  assumed  throughout  that  S always  gives  values  for  the 
q's  that  lie  in  the  unit  interval. 
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being  the  variance  of  each  value  stated  by  S.  The  improve- 
ment in  precision  due  to  coherence  is  only  appreciable  when 
n is  small;  for  n = 2 it  doubles,  as  we  saw  in  the  previous 
section.  The  form  of  (16)  is  worth  a comment  since  it 

generalizes.  The  adjustment,  u.-q. , of  the  stated  value  to 

^ ^ -1 

a reconciled  value  is  a multiple,  here  n , of  the  degree  of 
incoherence,  that  is,  how  far  Iq^  differs  from  its  coherent 
value,  1.  Here  each  q^  is  altered  by  the  same  amount. 

A subject  was  asked  to  assess  the  probabilities  that 
his  ultimate  death  would  fall  under  one  of  the  five  cate- 
gories: cancer,  heart  disease,  stroke,  other  natural, 

unnatural  (accident,  suicide,  etc.).  He  gave  the  values 

. 2 
given  in  the  second  column  of  Table  1.  The  total  is  0.83, 

exhibiting  a fair  degree  of  incoherence  perhaps  caused  by 

the  unpleasant  nature  of  the  events.  The  reconciled  values 

are  each  increased  by  0.17/5  = 0.034  and  are  given  in 

column  six. 

In  other  metrics  the  minimization  equations 
[F(q^)-F(^^)]F’  (^^)  = X 

in  Lagrange  form  are  non-linear.  Using  the  device  suggested 
above,  a first  guess  for  might  be  q^  with  the  result  that 

(18)  = q^-X/F' (q^)^. 


2 

The  last  column  gives  the  frequencies  (unknown  to  S)  for 
U.S.  residents.  These  values  are  not  to  be  interpreted  as 
the  correct  probabilities.  For  example,  a person  with  a 
known  history  of  heart  trouble  would  have  a higher  proba- 
bility than  the  frequency  value — a point  we  return  to  later. 
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so  that 


With  the  log-odds  metric  F'  (x)  = x”^(l-x) 

(19)  ^i  = ‘Ji  - 

providing  a larger  correction  when  q.  is  near  1/2  than  else- 
where. In  the  medical  example,  the  changes  upwards  are 
0.06,  0.04,  0.01,  0.04  and  0.01,  instead  of  a constant  0.034 
in  the  probability  metric.  The  reconciled  values  are  given 
in  the  penultimate  column  of  Table  4-1. 

The  reason  for  using  other  than  the  probability  metric 
is  the  fact  that  not  all  stated  probabilities  may  have  the 
same  errors  associated  with  them.  An  alternative  to  changing 
the  metric  is  for  either  S or  N to  provide  the  precision 
associated  with  each  value.  The  precision  of  an  assessment 
is  closely  related  to  its  stability,  i.e.,  the  degree  to 
which  it  varies  upon  further  reflection.  The  assessment  of 
the  probability  that  a coin  will  come  up  heads,  for  example, 
is  very  precise:  it  is  not  likely  to  depart  from  0.5.  The 
assessment  of  the  probability  that  a thumb-tack  will  fall  on 
its  head,  however,  is  much  less  precise.  Different  ways  of 
thinking  about  the  problem  could  yield  widely  different 
assessments . 

To  elicit  the  precision  associated  with  each  of  the 
q's,  we  asked  the  subject  to  quote  a range  of  values  for 
each  assessed  probability  (see  column  3 of  Table  4-1) . The 
quoted  ranges  were  interpreted  as  multiples  of  standard 

deviations  and  weights  used  inversely  proportional  to  the 

. 2 
variances  (column  4).  We  now  have  to  minimize  Ew^(q^-iT^) 

with  the  result  that 

(20)  TT.-q^  = (1  - i:q.)/w^IWj‘^  . 

The  results  for  the  medical  example  are  given  in  the  fifth 
column  of  Table  4-1. 
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4 . 2 Sub-partition;  Probability  Metric 

A problem  closely  related  to  that  described  in  the  pre- 
vious subsection  arises  where  S assesses  the  probabilities 
for  a number  of  exclusive,  but  not  exhaustive,  events 
...A^,  giving  values  ,q2 , . . .q^,  and  also  assesses  the 
probability  of  their  union,  citing  q'.  Incoherence  arises 
if  q'  ^ An  example  arises  in  the  same  medical  context 

where  S gave  values  for  four  natural  causes  of  death  (cancer, 
heart  disease,  stroke  and  other)  and  also  for  the  proba- 
bility of  his  dying  a natural  death.  With  = tt(A^)  and 
weights  w^,  the  problem  is  to  minimize, 

(21)  Iw^(q^-TT^)^  + w(q'-lTT^)2  . 

The  resulting  equations  are  linear  with  solutions 

(22)  ^i  " = (q'-^qj)/Wi{^Wj”^  + w“^}  . 

Notice  again  the  occurrence  of  the  measure  of  incoherence 
q'-Iq^.  If,  in  the  medical  example,  the  four  probabilities 
for  the  types  of  natural  death  had  been  as  before,  and  S 
had  quoted  0.92  as  the  probability  for  natural  death  (so 
that  he  was  coherent  with  his  original  0.08  for  unnatural 
death),  then  the  revision  is  exactly  as  before.  An  alterna- 
tive way  of  dealing  with  sub-partitions  using  conditional 
probabilities  is  described  below. 

Notice  that  in  both  these  subsections  it  has  been  assvuned 
that  the  correlations  between  S's  different  values  are  all 
zero.  The  most  likely  form  of  correlation  is  negative, 
the  subject  raising  one  probability  and  consequently  lowering 
another.  Indeed,  a coherent  S will  have  correlation  -n”^, 
giving  q'  = Iq^* 
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4 . 3 External  Conditioning;  Probability  Metric 

In  Section  2.2  we  mentioned  the  possibility  of  having 
some  target  events  whose  probabilities  are  required  and  that 
in  order  to  assess  these  values,  the  conversation  is  ex- 
tended to  other  events.  Here  we  consider  the  simplest  case 
of  a single  target  event  A and  the  extension  to  another 
event,  denoted  X to  distinguish  it  clearly  from  a target 
event.  Suppose  S is  sufficiently  conscious  of  coherence  so 
that  his  assessments  for  any  pair  of  complementary  events 
add  to  unity.  Suppose  further  that  S assesses  tt  (A)  , tt(aIx), 
ti(A|x),  and  tt  (X)  . Here,  the  opportunity  for  incoherence 
arises  since  tt  (A)  = TT(A|X)7r(X)  + Tr(A|X)iT(X)  may  not  hold 
for  the  corresponding  q's.  Our  hope  is  that  the  reconciled 
value  IT  (A)  will  be  an  improvement  over  the  raw  assessment 
q(A),  and  we  therefore  wish  to  evaluate  the  precision  of  the 
reconciled  value.  The  coherence  constraint  above  is  non- 
linear; to  avoid  this  let  us  suppose  7r(X)  is  known^  so  that 
q(X)  = IT  (X)  . We  later  generalize  to  the  case  where  this  is 
also  in  error.  The  process  will  be  called  external  con- 
ditioning, since  the  probability  of  the  target  event  is 
considered  conditional  on  some  external  event,  X. 

Since  no  new  difficulties  arise  by  generalizing,  we 
consider  a partition  into  Xj^,X2,...X^  and  not  just  X and  X, 
n = 2.  Write  q(A|S^)  = q^,  q (A)  = q',  the  it's  analogously 
and  q(X^)  = = ttCx^),  w'ith  Ik.  =1.  The  coherence  con- 

straint is  that  IT  (A)  = IiT  (A|  X^)  TT  (X^)  or  it'  = Itt^k^,  and  we 
have  to  minimize 


3 

An  example  is  where  X is  the  event  that  a team  wins  the 
toss  at  the  beginning  of  a contest.  It  would  then  generally 
be  agreed  that  q(X)  = Tr(X)  = p(X)  * 1/2.  The  event  A 
could  then  be  that  the  team  wins  the  contest. 
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(23) 


on  assuming  equal  weights  and  zero  correlations  for  the  q's. 


The  result/  for  the  target  event  A,  is  that 


(24) 


''  t I Zq  . K . -q ' 
it'  - q'  = 

7 

1 + Zk. 


In  words,  the  correction  to  q'  for  incoherence  is  the  de- 

2 

parture  from  coherence,  Zq^K^-q',  divided  by  (1+Zk^  ). 

This  divisor  is  the  variance  of  the  departure,  assuming 
all  q's  have  unit  variance.  The  precision  of  the  reconciled 
value  is  therefore  easily  found  to  be 


(25) 


2 -1 

1 + (^K^  ) 


times  the  precision  of  each  stated  value. 


There  is  another  simple  way  of  looking  at  tt  that  gener- 
alizes. In  effect,  S has  provided  two  values  for  tt  (A)  ; q’ 

. 2 
directly  and  Zq.K.  indirectly.  These  have  variances  o and 

2 2 ^ ^ 

Zk^  a , where  is  the  variance  of  each  statement  by  S. 
Taking  a weighted  average  of  these  two  values  with  weights 
inversely  proportional  to  the  variances,  we  obtain  tt  as 
given  by  (24) . 

Returning  to  a single  event  X,  the  case  n = 2,  the 

precision  of  tt  [A]  q (A)  ,q  (a|  X)  ,q  (A  [ X)  ] is  1 + [tt(X)^  + 

-2-1 

Tr(X)  ] relative  to  that  for  q(A).  The  improvement  is 
remarkable.  If  tt  (X)  = 1/2,  as  in  the  example  of  a toss, 
the  reconciled  value  has  three  times  the  precision  of  the 
original  value.  The  extension  of  the  conversation  to 
include  X has  tripled  the  precision,  and  it  would  appear 
that  the  extension  is  important  in  assess  probabilities. 

As  we  shall  see  below,  the  increase  in  precision  is 
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considerably  smaller  when  the  assessments  are  positively 
correlated,  as  is  likely  to  be  the  case  in  most  applications. 


It  is  easy  to  see  in  the  general  case  that  the  pre- 
cision (25)  is  maximized  when  all  the  k's  are  equal,  that 
is,  <^  = n it  is  best  to  choose  a partition  in  which 
the  constituent  events  are  all  equally  likely.  The  pre- 
cision for  the  reconciled  value  is  then  (n+1)  times  the 
precision  for  the  original  value.  (This  was  the  case  with 
the  example  of  winning  the  toss.)  Expressed  differently, 
the  increase  in  precision  due  to  refining  a partition  from 
n to  (n+1)  elements  is  equivalent  to  that  that  could  be  had 
from  an  independent  assessment  for  the  probability  of  the 
target  event.  Since  such  independence  judgments  are  not 
available  in  practice,  the  principle  of  extension  of  the 
conversation  can  be  used  to  play  a similar  role. 

The  result  that  among  partitions  of  size  n the  pre- 
cision is  maximized  when  all  probabilities  in  the  partition 
are  equal  is  a very  special  solution  to  what  we  call  the 
design  problem,  that  is,  the  problem  of  designing  questions 
to  be  answered  probabilistically  by  S in  such  a way  that  the 
expected  precision  of  the  reconciled  value  for  the  target 
event  is  maximized.  It  is  analogous  to  the  design  problem 
in  weighing  objects,  where  the  purpose  is  to  choose  the 
weighings  in  such  a way  as  to  maximize  the  precision  of  the 
final  determination  of  weight.  We  do  not  discuss  this 
problem  in  the  present  paper  except,  as  here,  when  solutions 
arise  as  a by-product  of  our  investigation  of  the  principle 
of  reconciliation.  One  cannot  choose  the  best  design  until 
the  reconciliation  problem  for  that  design  has  been  solved. 
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4 . 4 External  Conditioning ; Log-Odds  Metric 

With  this  metric  the  constraint  imposed  by  coherence  is 

non-linear  and  resort  has  to  be  made  to  devices  exemplified 

by  (15) . The  use  of  the  log-odds  metric  is  equivalent  to 

saying  that  the  variance  of  each  q is  proportional  to 
2 2 2 2 

IT  (1-t:)  , or  q (1-q)  approximately.  We  may  therefore 
replace  (23)  by  the  weighted  sum 

(26)  i:w^(q^-TT^)^  + w(q'-ZTT^<^)^, 

,-2 


-2 

W“[q'(l-q')]  . The  result  of  the  minimization  is  that 


in  generalization  of  (21),  with  w^  « and 

')]  The  result  of  tl 


(27) 


And  the  variance  is  proportional  to 

2\-l 


(28) 


1 /l 

-I- 

w w^  \ w '^i  / 


To  illustrate  the  reconciliation  procedure  for  external 
conditioning,  consider  the  assessment  of  the  probability  of 
an  energy  crisis  in  the  United  States  during  the  next  decade, 
denoted  A.  Let  X denote  the  development  of  new,  effective 
methods  for  the  use  of  solar  energy  during  the  next  decade. 
Suppose  q'=q(A)  = 0.7,  q^=q(AlX)  = 0.5,  q2=q(A|x)  = 0.8,  and 
K = q(X)  = IT  (X)  = 0.5.  With  the  probability  metric,  the 
reconciled  value  is  0.67  with  precision  three  times  that  of 
each  of  the  q's,  as  we  saw  in  Section  4.3.  With  the  log- 
odds  metric,  the  weights  are  proportional  to  Wj^  = .0625, 

W2  = .0256,  and  w = .0441.  The  reconciled  value  is  still 
0.67,  and  the  precision  is  slightly  more  than  three  times 
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that  of  the  original  value,  0.7.  This  suggests  that  the 
procedure  is  fairly  robust  to  changes  in  the  metric. 


4.5  External  Conditioninc 


Correlations 


When  S provides  q(A)  and  q(Alx),  he  will  often  find 
himself  considering  the  same  factors  in  both  assessments. 
Consequently,  it  may  be  reasonable  to  think  that  the  errors 
he  makes  in  these  two  assessments  are  correlated.  We  in- 
vestigate the  effect  of  this  on  external  conditioning  in  the 
special  case  of  a partition  into  X and  X,  where  q(A|X)  and 
q(AlX)  are  equally  correlated  with  q(A).  The  weights 
attached  to  the  two  conditional  assessments  are  also  sup- 
posed equal.  In  other  words,  the  situation  is  symmetric  in 
X and  X.  Only  the  probability  metric  is  considered. 


The  notation  for  the  variances  and  covariances  involved 
is  defined  by  giving  the  dispersion  matrix  of  the  assessed 
quantities,  other  than  it  (X)  = k which  we  are  still  assuming 
has  no  error.  This  is 


q'  = q(A) 

= q(Alx) 
= q{AlX) 

with  inverse 


(1-6M 

-pot (1-6) 

-pOT (1-6) 


-pat (1-6) 
a'  (1-pM 
a^  (p^-6) 


-pot  (1-6) 

(p^-6) 

0^  (1-pM 


divided  by  t ^ (1-6 ) (l+6-2p^ ) . It  is  this  matrix  that  pro- 
vides the  weights  to  be  used  in  the  quadratic  form  that  has 
to  be  minimized.  Thus  it  contains  a term 
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■IW  — 


-paT(l-6)  (q’-TT')  (q^-TT^)/o'T2  (1-6)  (lt-6-2pM 

corresponding  to  the  element  in  the  first  row  and  second 
column  of  the  matrix.  Here  tt  = + (1-k)it2  by  coherence. 

Tedious  calculation  shows  that 

^ [Kq.  + {1-K)q-  - q']a(a-pT) 

(29)  TT'  - q'  = 1 i 5 

Kt^(1-6)  + 6t  + a -2poT 

where  K = + (1-k)^.  With  correlations  p and  6 equal  to 

_2 

zero,  this  reduces  to  (27)  with  n = 2,  w^  = w„  = x and 

-2  ^ ^ 
w = a . It  may  also  be  obtained  by  taking  a weighted 

average  of  Kq^^  + (1-K)q2  and  q'  with  weights  that  depend 

on  the  variances  and  correlations.  The  variance  of  the 

reconciled  value  is  given  by 

var(^)=  oI^^[K(1-6)  (6-pM.3 

Kt^(1-6)  + 6t^  + a^-2paT 

Let  us  return  to  the  example  of  the  energy  crisis  where 

2 

q'  = 0.7,  q.  = 0.5,  and  q-  = 0.8,  with  k = 0.5,  and  a - 
2 ^ ^ 

T =1.  Ifp=6=0.5,  the  reconciled  value  is  0.67  exactly 

as  before,  but  the  variance  is  decreased  from  1.0,  say,  to 
0.67;  that  is,  the  precision  is  only  increased  by  50% 
rather  than  tripled  as  before.  This  result  reflects  a 
general  tendency  for  correlations  to  have  little  effect 
on  the  probabilities  but  a substantial  effect  on  the  pre- 
cisions . 

A comment  on  the  assessment  of  correlations  is  in 
order.  Recall  that  in  order  to  elicit  variances,  we  asked 
the  subject  to  set  upper  and  lower  limits  for  each  of  his 
q's,  and  treated  the  width  of  these  intervals  as  propor- 
tional to  the  respective  standard  deviations.  A similar 
procedure  can  perhaps  be  used  to  elicit  the  correlation 
between,  say,  q^^  and  q2 : the  subject  can  be  asked  to  set 
limits  for  q^^ — assuming  q2  is  fixed.  The  ratio  of  the  width 
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of  the  interval  for  when  q2  is  fixed  to  the  width  of  the 
same  interval  when  q2  is  allowed  to  vary,  is  a simple  trans- 
form of  the  correlation  coefficient.  Hov;ever,  this  task 
may  be  too  demanding,  and  it  appears  difficult,  although 
not  impossible,  to  elicit  from  subjects  meaningful  judgments 
of  correlations  between  assessments.  Fortunately,  the 
reconciled  probabilities  are  fairly  insensitive  to  the 
correlations,  which  affect  primarily  the  precisions.  Thus, 
instead  of  asking  the  subject  to  assess  the  correlations 
between  his  judgments,  we  could  ask  him  only  for  upper  and 
lower  limits  for  these  correlations.  These  values  could 
then  be  used  to  obtain  upper  and  lower  bounds  for  the  pre- 
cisions of  the  reconciled  probabilities. 

4 . 6 External  Conditioning;  Tr(X)  Assessed 

To  help  in  our  understanding  of  the  situation,  external 
conditioning  has  been  considered  only  when  the  probability 
of  the  conditioning  event  is  known,  since  this  makes  the 
analysis  linear.  We  now  pass  to  the  general  case  supposing 
that  all  four  probabilities  are  assessed  with  the  same 
errors  and  with  no  correlations.  As  before,  we  write  q{A)  = 
q' . q(A|X)  = q^^,  q(AlX)  = q2  and  introduce  q(X)  = q^. 

The  tt's  correspond  and  the  coherence  constraint  is  that 
Ti'=  + Tr2(l-Ti2).  The  function  to  be  minimized  is 

(30)  (q^-TT^)^  + (q2-^2^^  * 

It  is  possible  to  proceed  using  Taylor  series  expan- 
sion, but  in  this  case  a simpler  approximation  is  available 
using  the  alternative  way  of  interpreting  tt  ' in  (24)  as  a 
weighted  average.  Here  the  two  values  for  it  (A)  cited  by 
S are  q'  directly  and  q2^q3  + q2(l-q3)  indirectly.  The 
variance  of  the  former  is  o^,  say.  The  latter  is  non-linear, 
but  its  differential  is  q3<5qj^  + (l-q3)fiq2  + (q]^~q2)'^q3 
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2 2 2 

that  its  variance  is  approximately  ' 

times  a^.  The  evaluations  are  independent.  Taking  the 
weighted  average  and  rearranging  to  obtain  the  same  form  as 
before,  we  have 


(31) 


qiq-,  + q5(i-q-,)-q' 

TT*  - q'  = 

1 + q2  + + (qi-q2)^ 


The  precision  is 

(32)  1 + [q^  + (l-q3)^  + (q^-q2)^]~^ 


Both  these  results  are  approximate  but  probably  adequate 

for  most  applications.  The  result  for  the  precision  is 

especially  interesting  since  it  shows,  in  comparison  with 

the  result  for  knov/n  tt  (X)  , the  reduction  in  precision  due 

to  uncertainty  about  tt  (X)  . A straight  use  of  the  precision 

result  for  known  ti(X),  equation  (25),  would  give,  in  the 

2 2-1 

present  notation  1 + {q^  + (l-q^)  } , differing  from  (32) 

in  not  having  the  term  (qi~q2)^  brackets.  It  is  this 
term  that  produces  reduction  in  precision.  Hence,  lack  of 
knowledge  of  tt(X)  is  most  critical  when  q^^  q2;  that  is, 

q(A|x)  ^ q(A|x).  If  q^^  = then  it  does  not  matter. 

This  is  intuitively  sensible. 


In  the  energy  problem,  we  have  as  before:  q'  = 0.7, 
q^^  = 0.5,  q2  = 0.8,  and  q^  = 0.5,  except  that  q^  is  now 
uncertain.  The  reconciled  value  of  tt  is  0.67  as  before, 
illustrating  the  robustness  of  the  reconciliation  pro- 
cedure. The  precision  is  increased  by  the  external  con- 
ditioning from  1 to  2.70,  instead  of  3.0  as  before.  By 
using  the  full  equations  (30) , it  is  possible  to  calculate 
the  other  reconciled  values,  which  are  tt^^  = 0.52,  = 

0.82  and  tt^  = 0.49. 

The  extension  to  a general  partition  is  straightforward. 
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A special  case  of  external  conditioning  arises  when 
the  conditioning  event  is  a necessary  antecedant  to  the 
target  event.  For  example,  let  A be  the  event  that  S will 
die  from  cancer,  and  let  X be  the  event  that  S will  develop 
cancer  at  some  time  during  his  life.  A formal  way  of 
expressing  the  special  feature  of  this  situation  is  to  say 
tt{a|X)  = q(A|x)  = 0.  S will  then  provide  q'  = q(A),  q^^  = 
q(A|x)  and  q^  = q(X),  and  coherence  imposes  the  constraint 
TT  (A)  = TT  (A  I X)  IT  (X)  , or  IT ' = which  is  non-linear.  As 

before,  we  can  think  of  S as  providing  two  estimates  of 


tt':  q'  directly,  and  q^q,  indirectly.  If  the  former  has 
2 ^ 


variance  a , the  latter,  assum.ing  all  values  stated  by  S 


are  equally  precise  and  uncorrelated,  has  approximate 
2 2 2 

variance  (q^^  + q^)®  . Hence,  weighting  inversely  propor- 
tional to  the  variance,  we  have  for  a reconciled  value. 


the  result 


(33) 


q^qo  - q' 

’ - q*  = ^ ^ 

1 + q^  + q3 


2 2-1 

with  precision  1 + (q^^  + q^)  times  the  original  precision. 


Suppose  S assesses  the  probability  of  his  death  from 
cancer  as  q (A)  = 0.35,  the  probability  of  his  acquiring 
a cancer  as  q(X)  = 0.40,  and  finally  the  probability  of 
dying  from  an  acquired  cancer  as  q(A|X)  = 0.70.  With  q'  = 
0.35,  qj^  = 0.70  and  q^  = 0.40,  the  reconciled  value  is  0.31; 
and  the  precision  is  increased  2.54  times  its  original  value 


Notice  that  this  method  is  especially  valuable  when 
qj^  and  q^,  and  hence  q',  are  small,  since  then  the  increase 
in  precision  will  be  greatest.  However,  the  probability 
metric  may  not  be  too  appropriate  for  small  probabilities 
and  a log-odds  metric  preferred. 


4 . 8 Internal  Conditioning 


External  conditioning  deals  with  the  case  where  one 
extends  the  conversation  to  events  not  of  immediate  interest. 

If  there  is  more  than  one  target  event,  then  conditioning 
may  be  used  without  bringing  in  additional  events.  For 
example,  with  two  target  events  A and  B,  S may  provide 
probabilities  for  all  events  in  the  induced  partition,  AB, 

AB,  AB  and  AB.  Alternatively  S may  consider  A,  E condi- 
tional on  A,  and  finally  B conditional  on  A.  Of  course  the 
roles  of  A and  B may  be  reversed.  This  structure  is  called 
internal  conditioning  since  the  conditioning  events  are 
among  the  target  events.  Another  example  arises  with  a 
partition  A^^ , A2 , . . . A^^  of  the  type  considered  in  Section  4.1. 

The  probabilities  q(Aj^)  may  be  assessed  directly,  but  another 
possibility  is  to  consider  the  values  exemplified  by  q (A^^  | Aj^uA2)  • 
The  medical  example  (Table  4-1)  is  a case  in  point  where  S 
might  consider  the  probability  of  dying  from  cancer  con- 
ditional on  his  dying  a natural  death.  There  are  numerous 
possibilities  but  we  only  explore  in  detail  the  above  four- 
fold partition. 

Suppose  S is  coherent  in  that  his  stated  probabilities 
for  a partition  do  add  up  to  one,  but  they  may  exhibit  more 
subtle  types  of  incoherence.  He  can  then  assess  the  situa- 
tion for  two  events  A and  B by  first  giving  q(AB),  q(AB) 
and  q(AB) — when  q(AB)  will  be  one  minus  the  sum  of  these. 

He  can  then  look  at  the  situation  conditionally  using  q (A) , 
q(B|A)  and  q(BlA).  These  two  triples  provide  alternative 
complete  specifications  for  S's  opinions  about  the  two 
events.  The  coherence  constraints  such  as  ir  (AB)  = ti(A)tt(b|A) 

— there  are  three  of  them — are  non-linear,  and  a complete 
treatment  must  depend  on  numerical  procedures:  but  an 
approximate  argument  is  available.  This  utilizes  the  exter- 
nal conditioning  argument  of  Section  4.7,  treating  any 
member  of  the  partition,  AB  say,  as  having  a necessary 
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antecedant,  in  this  case  A.  Hence,  the  three  assessments 
q(AB),  q(A),  and  q(BlA)  can  be  taken  together  and  recon- 
ciled to  give,  equation  (33) , 

(341  S(AB)  - q(ABl  = • 

1 + q(AI^  + q(B|A)^ 

This  can  be  used  to  provide  reconciled  values  for  all  ele- 
ments of  the  partition.  Since  they  are  weighted  averages  of 
values  that  do  add  to  one  (by  assumption) , they  themselves 
will  also  add  to  one.  A reconciled  value  tt  (A)  may  be 
obtained  as  tt  (AB)  + tt(AB). 

If  S also  conditions  on  B and  provides  q(B),  q(A|B)  and 

q(A|B),  then  we  may  argue  as  follows:  S has  provided  three 

assessments  q(AB)  directly,  q(A)q(A|B)  and  q(B)q(A|B)  in- 

2 2 2 2 

directly.  These  have  variances  a , [q(A)  + q(B|A)  ]a  and 

2 2 2 

[q(B)  + q(B|A)  ]a  respectively,  the  last  two  being  approximate. 

They  may  be  combined  (Section  4.7)  as  a weighted  average 
with  weights  inversely  proportional  to  the  variances,  to 
give 

q(A)q(B|A)  q(B)q(A|B) 

q(AB)  + = _ 

q(A)^+q(B|A)"  + q (B) ^+q (A ] B) ^ 

(35)  Tr(AB)  = 

1 + [q(A)^+q(B|A)^]"^  + [q (B) ^+q (A | B) 

-2 

with  precision  given  by  the  denominator  times  o 

To  illustrate,  consider  the  results  of  a forthcoming 
election  in  a particular  district  in  the  United  States. 

Suppose  there  are  two  candidates  (a  Republican  and  a Demo- 
crat) for  the  Senate,  and  two  candidates  (a  Republican  and 
a Democrat)  for  the  House.  Let  A and  B,  respectively,  denote 
a democratic  victory  in  the  election  for  the  Senate  and  the 
House.  Suppose  further  that  some  political  observer  pro- 
vided the  assessments  displayed  in  the  upper  part  of  Table 
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4-2.  Thus  q(AB)  = 0.1,  q(B)  = 0.4,  q(A|B)  = 0.6,  etc.  As 
might  be  expected,  the  assessments  are  inconsistent.  The 
reconciled  values,  computed  according  to  equation  (34), 
using  either  A or  B as  the  conditioning  event,  are  displayed 
in  the  middle  part  of  the  table.  The  lower  part  of  Table 
4-2  presents  the  complete  reconciliation  based  on  all  the 
available  information  using  equation  (35);  it  also  presents 
the  precisions  of  the  reconciled  values  expressed  as  mul- 
tiples of  the  precisions  of  the  stated  values.  Thus,  tt(AE) 
0.129  with  a precision  7.9  times  that  of  the  original  value 
0.1.  The  reconciled  value  for  n (A)  will  be  0.129  + 0.340  = 
0.469.  It  is  not  possible  to  cite  the  precision  of  this 
value  since  the  two  values  that  have  been  combined  are  not 
independent;  the  sum  7.9  + 3.7  = 11.8  is  probably  not  too 
far  wide  of  the  mark. 


5.0  DISCUSSION 


In  his  treatise  on  the  foundations  of  statistics,  Savage 
writes : 

"According  to  the  personalistic  view,  the  role 
of  the  mathematical  theory  of  probability  is  to  enable 
the  person  using  it  to  detect  inconsistencies  in  his 
own  real  or  envisaged  behavior.  It  is  also  understood 
that,  having  detected  an  inconsistency,  he  will  re- 
move it.  An  inconsistency  is  typically  rem.ovable  in 
many  different  ways,  among  which  the  theory  gives  no 
guidance  for  choosing.  Silence  on  this  point  does 
not  seem  altogether  appropriate,  so  there  may  be  room 
to  improve  the  theory  here..." 

"To  approach  the  matter  in  a somewhat  different 
way,  there  seem  to  be  some  probability  relations  about 
which  we  feel  relatively  "sure"  as  compared  with 
others.  When  our  opinions,  as  reflected  in  real  or 
envisaged  action,  are  inconsistent,  we  sacrifice  the 
unsure  opinion  to  the  sure  ones.  The  notion  of  "sure" 
and  "unsure"  introduced  here  is  vague,  and  my  complaint 
is  precisely  that  neither  the  theory  of  personal 
probability,  as  it  is  developed  in  this  book,  nor 
any  other  device  known  to  me  renders  the  notion  less 
vague."  Savage  (1954,  pp.  57-58) 

The  present  paper  represents  an  attempt  to  deal  with 
the  issues  raised  by  Savage,  namely,  the  resolution  of  in- 
consistencies and  the  weighting  of  opinions.  Our  approach 
is  based  on  a division  of  labor  between  a fallible  subject  S 
and  a coherent  investigator  N who  uses  S's  assessments  to 
estimate  S's  "true"  probabilities  in  the  internal  method. 
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or  to  update  his  own  beliefs  about  the  world  in  the  external 
method.  Naturally,  such  an  approach  encounters  both  philo- 
sophical and  practical  difficulties. 

Perhaps  the  most  obvious  philosophical  objection  per- 
tains to  the  coherence  of  N.  Why  permit  first-order  inco- 
herence of  q but  exclude  second-order  incoherence  of  p?  If 
people  are  inevitably  fallible,  it  is  reasonable  to  postu- 
late a coherent  N?  Alternatively,  one  could  argue,  if  S has 
an  access  to  a coherent  N,  why  permit  inconsistency  in  the 
first  place? 

The  coherence  of  N is  needed  to  ensure  the  coherence  of 
the  reconciled  values.  If  the  core  distributions,  for 
example,  are  also  allowed  to  be  incoherent,  they  must  be 
reconciled  before  they  can  be  used  to  reconcile  the  basic 
assessments.  This  leads  to  an  infinite  regress  that  can  be 
avoided  only  by  assuming  coherence  somewhere  in  the  process. 
Indeed,  it  does  not  appear  unreasonable  to  assume  a fallible 
assessor  who  is  capable — in  a more  reflective  mood  and  per- 
haps with  the  help  of  paper  and  pencil — of  detecting  and 
reconciling  his  own  inconsi^, iencies . 

Notice  that  once  N's  coherent  assessments,  the  core 
distributions,  are  determined,  they  are  available  for  the 
resolution  of  any  problem  posed  by  S.  In  other  words,  the 
role  of  N is  to  link  together  the  possible  situations  that  S 
might  face.  This  is  in  the  true  spirit  of  coherence  in 
which  a problem  is  not  considered  in  isolation  but  viewed  in 
conjunction  with  other  problems,  both  real  and  conjectural. 

If  the  reader  considers  a simple  example,  he  can  easily 
conclude  that  our  methods  are  unnecessarily  involved — 
easier,  he  might  say  to  do  a naive  reconciliation  rather 
than  determine  the  core  distributions  and  then  use  these  to 
effect  the  reconciliation.  It  is  only  when  a set  of  examples 


is  considered  that  the  power  of  the  methods  is  revealed,  for 
then,  a simple  determination  of  core  elements  serves  for  the 
whole  set. 

The  present  approach  can  be  viewed  as  a compromise  be- 
tween two  extreme  positions  on  the  nature  of  probability 
assessments;  the  rationalistic  position  that  assumes  co- 
herence, and  the  empiricistic  position  that  denies  it. 

Neither  position  deals  with  the  reconciliation  problem;  the  ' 

former  effectively  ignores  the  issue,  while  the  latter  can-  j 

not  solve  it.  By  modeling  a person  as  composed  of  a fallible  j 

S and  a coherent  N,  we  attempt  to  provide  a more  realistic  ^ 

idealization  which  could,  nevertheless,  be  used  to  achieve 
rational  reconciliation. 

A central  feature  of  the  approach  developed  in  this 
paper  is  the  reliance  on  the  core  distributions  and  Bayes' 
rule  to  reconcile  incoherence.  Alternatively,  one  could 
start  by  considering  the  set  of  all  reconciled  values  and 
then  introduce  criteria  or  axioms  that  restrict  the  choice 
of  an  admissible  reconciliation.  For  example,  if  q(A)  = 

.62  and  q (A)  = .34,  one  may  wish  to  restrict  tt  so  that  .62 
S IT  (A)  < 1-.34.  Additional  constraints  could  further 
restrict  the  set  of  admissible  values.  This  approach 
represents  a viable  alternative  to  the  present  procedure. 

It  remains  to  be  seen,  however,  whether  one  could  develop 
a compelling  set  of  criteria  for  reconciliation  that  would 
lead  to  a unique,  or  at  least  a highly  constrained,  solution. 

I! 

> 

From  a practical  standpoint,  the  major  obstacle  to  the 
application  of  the  proposed  procedure  is  the  difficulty  in 
assessing  the  core  distributions.  All  three  distributions 
are  readily  interpretable:  plA)  is  N's  prior,  p(Tr|A) 
describes  the  relation  between  S's  true  beliefs  and  the 
external  world,  and  p(qlTT)  describes  the  relation  between 
S's  assessments  and  his  true  beliefs.  Although  these 
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expressions  are  psychologically  meaningful,  their  assessment 
may  prove  very  difficult  in  many  cases.  We  are  keenly 
aware  of  this  problem,  and  much  of  the  specific  assumptions 
discussed  in  Sections  3.0  and  4.0  were  introduced  to  simplify 
the  assessment  of  the  core  distributions.  It  remains  to  be 
demonstrated  that  this  information  can  be  elicited  from 
laymen  and/or  experts  for  non-trivial  problems. 

Reflection  suggests  to  us  that  the  introduction  of 
these  core  distributions,  and,  in  particular,  the  occurrence 
of  precisions  and  correlations,  is  a reasonable,  and  perhaps 
necessary,  requirement  in  the  real-world  situation  before 
reconciliation  is  possible. 

In  the  light  of  these  difficulties,  it  could  be  argued 
that  instead  of  applying  the  formal  procedures  developed  in 
this  paper,  we  could  simply  instruct  the  subject  to  resolve 
his  own  inconsistencies  in  the  way  that  he  finds  most  appro- 
priate. Although  this  approach  could  often  be  employed,  we 
believe  that  an  explicit  model  provides  a useful  tool  for 
the  analysis  of  coherence.  It  focuses  attention  on  the  data 
needed  to  resolve  incoherence,  and  it  provides  a rational 
procedure  for  reconciliation. 

Of  the  three  elements  in  the  core  distributions,  p(A)  and 
p(qlTr)  are  of  familiar  types,  but  the  third,  p(Tr|A)  is  rather 
novel.  It  provides,  together  with  p(Tr|A),  an  expression  of 
S's  ability  as  a probability  assessor;  and,  in  particular,  a 
statement  of  S's  variability.  In  effect,  it  looks  at  S as 
a diagnostic  instrument:  in  medical  terms,  if  the  patient 
has  appendicitis.  A,  what  probability  is  the  doctor,  S, 
going  to  assign  to  A;  and  similarly  for  a patient  with 
abdominal  pain  not  originating  from  a ruptured  appendix.  We 
have  seen  that  it  is  related  to  the  calibration  concept, 
ffAlu).  It  leads,  for  example  in  equation  (9),  to  unexpected 
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adaptations  from  it  (or  q)  to  revised  probabilities  for  A, 
P{a|it)  or  p(A|q),  which  cannot  always  be  it  or  q. 

Simon  French  (1977)  has  suggested  to  us  that  p(Tr|A) 
might  depend  on  p(A).  In  words,  N's  assessment  of  S might 
depend  on  his  own  views  about  A,  the  event  under  discussion. 
He  cites  the  example  where  past  experience  has  shown  that  in 
similar  situations,  N and  S have  tended  to  agree:  then 
P(tt1a)  might  peak  around  tt  = p(A).  If  this  is  admitted,  we 
are  lead  to  a curious  probability,  p(tt1a,p(A))  and  an  unex- 
pected form  of  Bayes  Theorem  that  French  has  considered, 
namely 


p(A|iT,p)  = p(tt|a,p)p(A) 

where  p = p(A).  This  is  an  interesting  idea  that  introduces 
a reasonable  correlation  between  N and  S,  but  it  should  be 
noted  that  our  model  partly  allows  for  such  correlation.  We 
suppose  TT  is  unaffected  by  p,  given  A and  given  A,  but  this 
will  not  imply  that  ti  is  unconditionally  free  of  p;  indeed, 
quite  the  contrary  if  S is  a good  appraiser. 

There  are  at  least  two  ways  in  which  the  model  can  be 
generalized.  First,  the  world  external  to  both  N and  S can 
contain  uncertain  quantities  and  not  just  uncertain  events. 
This  has  been  discussed  by  Morris  (1574,  1977).  An  example 
might  be  the  case  where  the  meteorologist  is  forecasting  the 
amount  of  rain  tomorrow  rather  than  whether  or  not  it  will 
rain. 


A second  generalization  is  to  the  case  where  the  data- 
base for  S changes.  In  our  discussion  it  has  been  supposed 
fixed.  An  exam.ple  arises  when  the  meteorologist,  forecasting 
the  weather  on  day  2 learns  about  what  the  weather  has  been 
on  day  1.  The  role  of  the  data-base  certainly  needs  more 
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examination:  it  might  resolve  the  dilemma  proposed  by  French, 
in  that  the  data  common  to  N and  S might  explain  the  possible 
correlation  suggested  by  him. 
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